How can my casual alcohol use affect my intelligence?

Yes, but not dramatically. Alcohol is a central nervous system depressant. Part of that depressant effects does, under some limited cirstances, alcohol decrease stress but it also decreases the functioning of various areas of the brain, including those involved in memory. Minor memory lapses while intoxicated are now believed to be mini-blackouts. Major memory loss with more regular larger doses is what causes full blown blackouts (loss of memory of events even though you were conscious and functioning during the events). However, these are initially only acute effects, meaning the effect (whatever it is) is only present when you are intoxicated and there is little or no measurable carry-over to when you are sober. Unfortunately, that does change with chronic use in which the amount you drink slowly increases because you acquire tolerance and because the alcohol is causing longer lasting changes to the way brain cells communicate. Eventually (usually after many months or years) the alcohol itself acts as a stressor as does not having it. So, initially minor intelligence changes when you are under the influence (of 2-3 shots), impaired driving, etc., but more insideous problems with longer term use.