Cannabinoid signaling is involved in different brain functions and it is mediated by the cannabinoid receptor 1 (CNR1), which is encoded by the CNR1 gene. Previous evidence suggests an association between cognition and cannabis use. The logical interaction between genetically determined cannabinoid signaling and cannabis use has not been determined. Therefore, we investigated whether CNR1 variation predicts CNR1 prefrontal mRNA expression in postmortem prefrontal human tissue. Then, we studied whether functional variation in CNR1 and cannabis exposure interact in modulating prefrontal function and related behavior during working memory processing. Thus, 208 healthy subjects (113 males) were genotyped for the relevant functional SNP and were evaluated for cannabis use by the Cannabis Experience Questionnaire. All individuals performed the 2-back working memory task during functional magnetic resonance imaging. CNR1 rs1406977 was associated with prefrontal mRNA and individuals carrying a G allele had reduced CNR1 prefrontal mRNA levels compared with AA subjects. Moreover, functional connectivity MRI demonstrated that G carriers who were also cannabis users had greater functional connectivity in the left ventrolateral prefrontal cortex and reduced working memory behavioral accuracy during the 2-back task compared with the other groups. Overall, our results indicate that the deleterious effects of cannabis use are more evident on a specific genetic background related to its receptor expression.