Male gender is a risk factor for the development of hepatocellular carcinoma (HCC) but the mechanisms are not fully understood. The RNA binding motif gene on the Y chromosome (RBMY), encoding a male germ cell-specific RNA splicing regulator during spermatogenesis, is aberrantly activated in human male liver cancers. This study investigated the in vitro oncogenic effect and the possible mechanism of RBMY in human hepatoma cell line HepG2 and its in vivo effect with regards to the livers of human and transgenic mice. RBMY expression in HepG2 cells was knocked down by RNA interference and the cancer cell phenotype was characterized by soft-agar colony formation and sensitivity to hydrogen-peroxide-induced apoptosis. The results revealed that RBMY knockdown reduced the transformation and anti-apoptotic efficiency of HepG2 cells. The expression of RBMY, androgen receptor (AR) and its inhibitory variant AR45, AR-targeted genes insulin-like growth factor 1 (IGF-1) and insulin-like growth factor binding protein 3 (IGFBP-3) was analyzed by quantitative RT-PCR. Up-regulation of AR45 variant and reduction of IGF-1 and IGFBP-3 expression was only detected in RBMY knockdown cells. Moreover, RBMY positive human male HCC expressed lower level of AR45 as compared to RBMY negative HCC tissues. The oncogenic properties of RBMY were further assessed in a transgenic mouse model. Liver-specific RBMY transgenic mice developed hepatic pre-cancerous lesions, adenoma, and HCC. RBMY also accelerated chemical carcinogen-induced hepatocarcinogenesis in transgenic mice. Collectively, these findings suggest that Y chromosome-specific RBMY is likely involved in the regulation of androgen receptor activity and contributes to male predominance of HCC.