High levels of insulin-like growth factor-1 (IGF-1) have been associated with increased risk of developing several types of cancer including breast cancer. A set of nine haplotype tagging SNPs (htSNPs) in the IGF1 gene were associated with IGF-1 levels and prostate cancer in a Swedish population. We aimed to study the nine htSNPs in three haplotype blocks (block1: rs855211, rs35765, rs2162679; block2: rs1019731, rs7956547, rs5742632; and block3 rs2033178, rs7136446, rs6220) combined into diplotypes, and three additional SNPs (rs5742612, rs35765817, rs35455143) in relation to IGF-1 levels, BRCA status, the IGF1 CA-repeat microsatellite, and breast cancer in a population of 325 Swedish women from breast cancer high-risk families. Questionnaire data and blood samples for IGF-1 and genetic analyses were obtained twice during the menstrual cycle from 269 women aged 40 years or younger. SNP analyses were also performed in 56 BRCA1/2 mutation carriers. Women (n = 14) with any rare variant block1 diplotype had higher odds to be BRCA1 mutation carriers OR 4.1 (95% CI 1.4-12.2), to lack the common IGF1 19 CA-repeat allele OR 33.3 (95% CI 6.6-166.7), and were more likely to develop early-onset breast cancer (Log Rank P < 0.001) than women with common block1 diplotypes. In the subgroup of BRCA1 mutation carriers, block1 rare diplotypes were associated with earlier diagnosis (Log Rank P = 0.031). No association was found between IGF-1 levels and individual SNPs or diplotypes. If confirmed, these rare diplotypes may identify women with particularly high risk for early-onset breast cancer and this group should be included in forthcoming studies.