Lexical-semantic competency in mild cognitive impairment (MCI) ε4 carriers was used as an endophenotype, and gray matter volume in MCI ε4 carriers/noncarriers and in noncarrier controls was compared. Residual gray matter volumes were correlated with age of acquisition values for words from a category fluency task, an index of semantic competency. MCI patients had significantly impoverished lexical-semantic output compared with controls, more marked in MCI ε4 carriers. Smaller volumes in the left hippocampus, bilateral regions of the uncus, and posterior cingulate cortex were associated with a tendency to retrieve earlier acquired words in the category fluency task in MCI ε4 carriers, whereas poor semantic performance in MCI noncarriers was associated with smaller volumes in the left uncus, bilateral regions of the parahippocampal gyrus, and hippocampus, and also in a large number of neocortical regions. There was a significant semantic competency by genotype interaction in the left perirhinal cortex, in a number of left frontal and temporal areas and in the right inferior parietal lobule and precuneus. MCI ε4 carriers, when compared with noncarriers, had lower gray matter volume values confined to the right precuneus and the cerebellum bilaterally, but the converse comparison showed that MCI noncarriers had lower values in extensive frontal, temporal, and parietal regions of the neocortex. Similar brain volumetric variations linked to genotype were found in minimal-to-mild AD. The results suggest a relatively specific impact of apolipoprotein E (APOE) ε4 burden and underline the value of linguistic assessment in preclinical diagnosis.