Vascular factors have been shown to affect the rate of Alzheimer's disease (AD) progression. However, the effect of the APOE ε4 allele on rate of progression has been ambiguous. Little research to date has examined an interaction between vascular factors and the APOE ε4 allele in predicting decline among AD patients. 216 participants with incident AD from a population of elderly persons in Cache County, Utah, were followed for a mean of 3.3 years and 4.2 follow-up visits. A history of vascular risk factors and conditions and anti-hypertensive use was assessed at the diagnostic visit. Linear mixed effects models tested interactions between the vascular factors, APOE ε4, and time as predictors of clinical progression on the Mini-Mental State Exam (MMSE) and Clinical Dementia Rating-Sum of Boxes (CDR-SB). Multiple comparisons were corrected using the Holm-Bonferroni method. There was a 3-way interaction between stroke, APOE ε4 and time in predicting MMSE decline (LR χ² = 10.32, 2 df, p = 0.006). For the CDR-SB, there were 3-way interactions between the APOE ε4, time and either myocardial infarction (LR χ² = 17.83, 2 df, p = 0.0001) or stroke (LR χ² = 11.48, 2 df, p = 0.003. Results suggest a complex relationship between the APOE ε4 and vascular factors in predicting cognitive and functional progression. Among individuals with a history of stroke or myocardial infarction at baseline, progression of AD is influenced by APOE ε4 carrier status and varies by time after AD diagnosis.