A common variant (rs53576) of the OXTR gene has been implicated in a number of socio-emotional phenotypes, such as anxiety-related behavior. Previous studies have demonstrated that A-allele carriers have higher levels of physiological and dispositional stress reactivity and depressive symptomatology compared to those with the GG genotype, but the mediating neural mechanisms remain poorly understood. We combined voxel-based morphometry and resting-state functional connectivity analyses in a large cohort of healthy young Chinese Han individuals to test the hypothesis that the OXTR gene polymorphism influences an anxiety-related temperamental trait, as assessed by the harm avoidance subscale from the Tridimensional Personality Questionnaire via modulating the gray matter volume and resting-state functional connectivity of the brain, especially the limbic system. We revealed that female subjects with the AA genotype showed increased harm avoidance scores relative to G-carrier females. We also found that, compared to female individuals with the GG/GA genotype, female individuals with the AA genotype exhibited significantly smaller amygdala volumes bilaterally (especially the centromedial subregion), with a trend of allele-load-dependence. Compared to female individuals with the GG/GA genotype, female subjects with the AA genotype demonstrated reduced resting-state functional coupling between the prefrontal cortex and amygdala bilaterally, also with an allele-load-dependent trend. Furthermore, the magnitude of prefrontal-amygdala coupling in the left hemisphere was positively correlated with harm avoidance scores in female subjects. Our findings highlight a possible neural pathway by which a naturally occurring variation of the OXTR gene may affect an anxiety-related temperamental trait in female subjects by modulating prefrontal-amygdala functional connectivity.