Alzheimer's disease (AD) is an increasingly prevalent, fatal neurodegenerative disease that has proven resistant, thus far, to all attempts to prevent it, forestall it, or slow its progression. The ε4 allele of the Apolipoprotein E gene (APOE4) is a potent genetic risk factor for sporadic and late-onset familial AD. While the link between APOE4 and AD is strong, many expected effects, like increasing the risk of conversion from MCI to AD, have not been widely replicable. One critical, and commonly overlooked, feature of the APOE4 link to AD is that several lines of evidence suggest it is far more pronounced in women than in men. Here we review previous literature on the APOE4 by gender interaction with a particular focus on imaging-related studies.