The apolipoprotein E-ε4 allele is a well-known genetic risk factor for late-onset Alzheimer's disease, which also impacts the cognitive functions and brain network connectivity in healthy middle-aged adults without dementia. Previous studies mainly focused on the effects of apolipoprotein E-ε4 allele on single index using task or resting-state fMRI. However, how these evoked and spontaneous BOLD indices interact with each other remains largely unknown. Therefore, we evaluated the 'rest-stimulus interaction' between working-memory activation and resting-state connectivity in middle-aged apolipoprotein E-ε4 carriers (n=9) and non-carriers (n=8). Four n-back task scans (n = 0, 1, 2, 3) and one resting-state scan were acquired at a 3T clinical MRI scanner. The working-memory beta maps of low-, moderate-, and high-memory loads and resting-state connectivity maps of default mode, executive control, and hippocampal networks were derived and compared between groups. Apolipoprotein E-ε4 carriers presented declined working-memory activation in the high-memory load across whole brain regions and reduced hippocampal connectivity compared with non-carriers. In addition, disrupted rest-stimulus interactions were found in the right anterior insula and bilateral parahippocampal regions for middle-aged adults with apolipoprotein E-ε4 allele. The rest-stimulus interaction improved the detectability of network integrity changes in apolipoprotein E-ε4 carriers, demonstrating the disrupted intrinsic connectivity within the executive-functional regions and the modulated memory-encoding capability within hippocampus-related regions.