We investigated the effects of the brain-derived neurotrophic factor (BDNF) Val66Met polymorphism on age-associated changes of brain morphology in 109 Japanese healthy subjects using MRI with optimized voxel-based morphometry technique. A significant age-related volume reduction was found in the dorsolateral prefrontal cortices (DLPFC), anterior cingulate cortices, and temporal and parietal cortices in all subjects. Further analysis revealed a significantly negative correlation between age and the volume of the bilateral DLPFC only in the Met-BDNF carriers, and a significant interaction between the polymorphism and age-associated volume changes in the bilateral DLPFC. Furthermore, Met-carriers showed a significant interaction (p<0.0001) between the gender and the genotype on the gray matter volume in the DLPFC, and female Met-carriers showed more widespread age-associated volume reduction in DLPFC than male Met-carriers. Our data suggest that the Val66Met polymorphism may impact on age-related changes of the brain, which might be associated with the functional variance of neuroprotective effects of the BDNF. Furthermore, we suggest that genotype effects of the BDNF gene on brain morphology might differ in female from in male.