Sex-determining region Y-related high mobility group box 4 (SOX4) has been proven to serve as a critical role in cancer development and progression. However, little is known about the pathological role of SOX4 in breast cancer patients. The purpose of this study is to measure the expression of SOX4 in breast cancer patients and to explore the clinical significance of SOX4. Using RT-PCR and Western blot, messenger RNA (mRNA) and protein expression of SOX4 were measured in breast cancer tissues and adjacent normal mammary tissues. The relationship of SOX4 expression with clinical characteristics of 148 breast cancer patients was analyzed by immunohistochemistry. In the present study, our results indicated that SOX4 mRNA and protein were highly expressed in breast cancer tissues compared with adjacent normal mammary tissues and positively correlated with clinical stage (I-II vs. III-IV; P = 0.008), T classification (T1-T2 vs. T3-T4; P = 0.013), N classification (N0-N1 vs. N2-N3; P < 0.001), M classification (M0 vs. M1; P = 0.001), estrogen receptor (negative vs. positive; P = 0.029), progesterone receptor (negative vs. positive; P = 0.004), and histological grade (G1 vs. G2-G3; P = 0.033) in breast cancer patients. Furthermore, we also found that SOX4 protein overexpression was an unfavorable prognostic factor in breast cancer patients (P < 0.001), regardless of clinical stage, tumor size, lymph node metastasis, and distant metastasis. Finally, high SOX4 expression was an independent poor prognostic factor for pancreatic patients through multivariate analysis (P = 0.033). In conclusion, SOX4 overexpression serves as an unfavorable prognostic biomarker in breast cancer patients.