A cross talk between the Estrogen Receptor (ESR1) and the Retinoblastoma (pRb) pathway has been demonstrated to influence the therapeutic response of breast cancer patients but the full mechanism remains poorly understood. Here we show that the N-terminal domain of pRb interacts with the CD domain of ESR1 to allow for the assembly of intermediate complex chaperone proteins HSP90 and p23. We demonstrated that a loss of pRb in human/mouse breast cells decreases the expression of the ESR1 protein through the proteasome pathway. Our work reveals a novel regulatory mechanism of ESR1 basal turnover and activity and an unanticipated relationship with the pRb tumor suppressor.