First-degree family history of sporadic Alzheimer disease (AD) and the apolipoprotein E epsilon4 (APOE4) are risk factors for developing AD. Although the role of APOE4 in AD pathogenesis has been well studied, family history remains a rarely studied and poorly understood risk factor. Both putatively cause early brain changes before symptomatic disease, but the relative contribution of each to brain function is unknown. We examined 68 middle-aged participants with a parent diagnosed with AD [family history (+FH)] and 64 age- and education-matched controls without a first-degree family history of any dementia [no family history (-FH)]. All underwent cognitive testing, APOE genotyping, and a functional magnetic resonance imaging encoding task that required discrimination of novel items from previously learned items. A 2 x 2 factorial ANOVA (presence/absence of parental family history and presence/absence of the APOE4) was used to detect group effects. A greater response to novel items was detected in the mesial temporal lobe and fusiform gyrus bilaterally among persons without a first-degree family history of AD. In hippocampal areas, the -FH +epsilon4 group exhibited the greatest signal change, and the +FH +epsilon4 group exhibited the least. These findings indicate that FH of AD is an important predictor of hippocampal activation during encoding and that FH may modulate the effect of APOE4 in these middle-aged adults, suggesting that an as yet unspecified factor embodied in first-degree family history of AD is influencing the expression of APOE4 on brain function.