The international American Joint Committee on Cancer/Union for International Cancer Control (AJCC/UICC) tumour-node-metastasis (TNM) staging system provides the current guidelines for the classification of cancer. However, among patients within the same stage, the clinical outcome can be very different. More recently, a novel definition of cancer has emerged, implicating at all stages a complex and dynamic interaction between tumour cells and the immune system. This has enabled the definition of the immune contexture, representing the pre-existing immune parameters associated with patient survival. Even so, the role of distinct immune cell types in modulating cancer progression is increasingly emerging. An immune-based assay named the 'Immunoscore' was defined to quantify the in situ T cell infiltrate and was demonstrated to be superior to the AJCC/UICC TNM classification for patients with colorectal cancer. This Review provides a broad overview of the main immune parameters positively or negatively shaping cancer development, including the Immunoscore, and their prognostic and predictive value. The importance of the immune system in cancer control is demonstrated by the requirement for a pre-existing intratumour adaptive immune response for effective immunotherapies, such as checkpoint inhibitors. Finally, we discuss how the combination of multiple immune parameters, rather than individual ones, might increase prognostic and/or predictive power.