Many epithelial carcinomas, including ovarian, are refractory to the antiproliferative effects of transforming growth factor (TGF) beta. In some cancers, TGF-beta resistance has been linked to TGF-beta receptor II (TbetaR-II) and Smad4 mutations; however, in ovarian cancer, the mechanism of resistance remains unclear. Primary ovarian epithelial cell cultures were used as a model system to determine the mechanisms of TGF-beta resistance. To simulate in vivo responses to TGF-beta, primary cultures derived from normal human ovarian surface epithelium (HOSE) and from ovarian carcinomas (CSOC) were grown on collagen I gel, the predominant matrix molecule in the ovarian tumor milieu. When treated with 5 ng/ml TGF-beta for 72 h, HOSE (n = 11) proliferation was inhibited by 20 +/- 21% on average. In contrast, CSOC (n = 10) proliferation was stimulated 5 +/- 10% in response to TGF-beta (a statistically significant difference in response when compared with HOSE; P = 0.001). To dissect the TGF-beta/Smad signaling pathway we used a quantitative RNase protection assay (RPA) for measuring mRNA levels of TGF-beta pathway components in 20 HOSE and 20 CSOC cultures. Basal mRNA levels of TGF-beta receptors I and II, downstream signaling components Smad2, 3, 4, 6, 7, and the transcriptional corepressors Ski and SnoN did not show a statistically significant difference between HOSE and CSOC, and cannot explain their differential susceptibility to TGF-beta-induced cell cycle arrest. To assess functional differences of the TGF-beta pathway in TGF-beta-sensitive HOSE and TGF-beta-resistant CSOC, we measured Smad2/4 and 3/4 complex induction after TGF-beta treatment. HOSE and CSOC showed equivalent Smad2/4 and 3/4 complex induction after TGF-beta exposure for 0, 0.5, 2, and 4 h. It has been proposed that SnoN and Ski are corepressors of the TGF-beta/Smad pathway and undergo TGF-beta-induced degradation followed by reinduction of SnoN mRNA. However, our data show equivalent SnoN degradation in HOSE and CSOC, and equivalent SnoN mRNA induction after TGF-beta treatment. Surprising, TGF-beta-induced Ski degradation was not observed in HOSE or CSOC, suggesting that Ski may not function as a TGF-beta/Smad corepressor in ovarian epithelial cells. These data implied that the TGF-beta/Smad pathway remains functional in CSOC, although CSOC cells are resistant to antimitogenic TGF-beta effects. CSOC resistance to TGF-beta coincided with the loss of c-myc down-regulation. These data suggest that TGF-beta/Smad signaling is blocked downstream of Smad complex formation or that an alternate signaling pathway other than TGF-beta/Smad may transmit TGF-beta-induced cell cycle arrest in the ovarian epithelium.