Constitutive activation of Signal Transducers and Activators of Transcription 3 (STAT3) signaling is frequently detected in breast and pancreatic cancer. Inhibiting constitutive STAT3 signaling represents a promising molecular target for therapeutic approach. Using structure-based design, we developed a non-peptide cell-permeable, small molecule, termed as XZH-5, which targeted STAT3 phosphorylation. XZH-5 was found to inhibit STAT3 phosphorylation (Tyr705) and induce apoptosis in human breast and pancreatic cancer cell lines expressing elevated levels of phosphorylated STAT3. XZH-5 could also inhibit interleukin-6-induced STAT3 phosphorylation in cancer cell lines expressing low phosphorylated STAT3. Inhibition of STAT3 signaling by XZH-5 was confirmed by the down-regulation of downstream targets of STAT3, such as Cyclin D1, Bcl-2, and Survivin at mRNA level. In addition, XZH-5 inhibited colony formation, cell migration, and enhanced the cytotoxicity of chemotherapeutic drugs when combined with Doxorubicin or Gemcitabine. Our results indicate that XZH-5 may be a potential therapeutic agent for breast and pancreatic cancers with constitutive STAT3 signaling.