Accumulation of β-amyloid (Aβ) in the brain is associated with memory decline in healthy individuals as a prelude to Alzheimer's disease (AD). Genetic factors may moderate this decline. We examined the role of apolipoprotein E (ɛ4 carrier[ɛ4(+)], ɛ4 non-carrier[ɛ4(-)]) and brain-derived neurotrophic factor (BDNF(Val/Val), BDNF(Met)) in the extent to which they moderate Aβ-related memory decline. Healthy adults (n=333, Mage=70 years) enrolled in the Australian Imaging, Biomarkers and Lifestyle study underwent Aβ neuroimaging. Neuropsychological assessments were conducted at baseline, 18-, 36- and 54-month follow-ups. Aβ positron emission tomography neuroimaging was used to classify participants as Aβ(-) or Aβ(+). Relative to Aβ(-)ɛ4(-), Aβ(+)ɛ4(+) individuals showed significantly faster rates of cognitive decline over 54 months across all domains (d=0.40-1.22), while Aβ(+)ɛ4(-) individuals showed significantly faster decline only on verbal episodic memory (EM). There were no differences in rates of cognitive change between Aβ(-)ɛ4(-) and Aβ(-)ɛ4(+) groups. Among Aβ(+) individuals, ɛ4(+)/BDNF(Met) participants showed a significantly faster rate of decline on verbal and visual EM, and language over 54 months compared with ɛ4(-)/BDNF(Val/Val) participants (d=0.90-1.02). At least two genetic loci affect the rate of Aβ-related cognitive decline. Aβ(+)ɛ4(+)/BDNF(Met) individuals can expect to show clinically significant memory impairment after 3 years, whereas Aβ(+)ɛ4(+)/BDNF(Val/Val) individuals can expect a similar degree of impairment after 10 years. Little decline over 54 months was observed in the Aβ(-) and Aβ(+) ɛ4(-) groups, irrespective of BDNF status. These data raise important prognostic issues in managing preclinical AD, and should be considered in designing secondary preventative clinical trials.