Testosterone therapy (TT) is administered to enhance masculinization in transgender individuals. The long-term effect of exogenous testosterone on breast tissues remains unclear. Our study evaluated the modulation of breast morphology by TT in transgender individuals with special attention to duration of TT. We reviewed 447 breast surgical specimens from gender affirming chest-contouring surgery, and compared histopathological findings including degree of lobular atrophy, and atypical and non-atypical proliferations between subjects who did (n = 367) and did not (n = 79) receive TT. TT for one patient was unknown. TT for >12 months was associated with seven histopathological features. Longer duration of TT was significantly associated with higher degrees of lobular atrophy (p < 0.001). This relationship remained significant after accounting for age at surgery, ethnicity, body mass index, and presurgical oophorectomy (adjusted p < 0.001). Four types of lesions were more likely to be absent in breast tissues exposed to longer durations of TT: cysts (median = 16.2 months; p < 0.01; adjusted p = 0.01), fibroadenoma (median = 14.8 months; p = 0.02; adjusted p = 0.07), pseudoangiomatous stromal hyperplasia (median = 17.0 months; p < 0.001; adjusted p < 0.001), and papillomas (median = 14.7 months; p = 0.04; adjusted p = 0.20). Columnar cell change and mild inflammation were also less likely to occur in subjects receiving TT (p < 0.05), but were not linked to the duration of TT. Atypia and ductal carcinoma in situ were detected in 11 subjects (2.5%) all of whom received TT ranging from 10.1 to 64.1 months. The incidental findings of high-risk lesions and carcinoma as well as the risk of cancer in residual breast tissue after chest-contouring surgery warrant the consideration of culturally sensitive routine breast cancer screening protocols for transgender men and masculine-centered gender nonconforming individuals. Long-term follow-up studies and molecular investigations are needed to understand the breast cancer risk of transgender individuals who receive TT.