Levels of enzymes that determine testosterone catabolism such as CYP3A4 have been associated with prostate cancer (PCa) risk. Although some studies have related CYP3A4*1B allele, a gene polymorphism that modifies CYP3A4 expression level, with PCa risk, others have failed, suggesting that additional genetic variants may be involved. Expression of CYP3A4 is largely due to the activation of Pregnane X Receptor (PXR). Particularly, rs2472677 and rs7643645 PXR polymorphisms modify CYP3A4 expression levels. To evaluate whether PXR-HNF3β/T (rs2472677), PXR-HNF4/G (rs7643645), and CYP3A4*1B (rs2740574) polymorphisms are associated with PCa a case control-study was performed. The multiple testing analysis showed that the PXR-HNF4/G polymorphism was associated with higher levels of prostate-specific antigen (PSA) in patients with PCa (OR = 3.99, p = 0.03). This association was stronger in patients diagnosed at the age of 65 years or older (OR = 10.8, p = 0.006). Although the CYP3A4*1B/*1B genotype was overrepresented in PCa patients, no differences were observed in the frequency of this and PXR-HNF3β/T alleles between controls and cases. Moreover, no significant association was found between these polymorphisms and PSA, Gleason grade, or tumor lymph node metastasis.