Kaiso is a BTB/POZ transcription factor that is ubiquitously expressed in multiple cell types and functions as a transcriptional repressor and activator. Little is known about Kaiso expression and localization in breast cancer. Here, we have related pathological features and molecular subtypes to Kaiso expression in 477 cases of human invasive breast cancer. Nuclear Kaiso was predominantly found in invasive ductal carcinoma (IDC) (p = 0.007), while cytoplasmic Kaiso expression was linked to invasive lobular carcinoma (ILC) (p = 0.006). Although cytoplasmic Kaiso did not correlate to clinicopathological features, we found a significant correlation between nuclear Kaiso, high histological grade (p = 0.023), ERα negativity (p = 0.001), and the HER2-driven and basal/triple-negative breast cancers (p = 0.018). Interestingly, nuclear Kaiso was also abundant in BRCA1-associated breast cancer (p<0.001) and invasive breast cancer overexpressing EGFR (p = 0.019). We observed a correlation between nuclear Kaiso and membrane-localized E-cadherin and p120-catenin (p120) (p<0.01). In contrast, cytoplasmic p120 strongly correlated with loss of E-cadherin and low nuclear Kaiso (p = 0.005). We could confirm these findings in human ILC cells and cell lines derived from conditional mouse models of ILC. Moreover, we present functional data that substantiate a mechanism whereby E-cadherin controls p120-mediated relief of Kaiso-dependent gene repression. In conclusion, our data indicate that nuclear Kaiso is common in clinically aggressive ductal breast cancer, while cytoplasmic Kaiso and a p120-mediated relief of Kaiso-dependent transcriptional repression characterize ILC.