Drinking and Memory Loss: Is it Normal?

About a year ago, my neurologist told me I can drink, but to stop after two drinks. So that’s what I did for six months. Then when I saw him again, I told him, and he was pretty happy about that. But then after that, I started drinking more. 3-5 drinks at once. So not too horrible but still more than he suggested. Anyway, one thing I’ve noticed is that the more I drink, the LESS likely I am to black out. Five drinks might give me a hangover the next day, but at least I remember everything. But if I only have 2-3 drinks, while I may not be hung over the next day, I remember very little about the last night. It’s weird. I would expect the opposite. Is this normal, or could it maybe be related to my meds?