Background Breastfeeding has been related to better cognitive development in children. However, due to methodological challenges, such as confounding, recall bias or insufficient power, the mechanism and nature of the relation remains subject to debate. Methods We included 3761 participants of a population-based cohort study from fetal life onwards and examined the association of breastfeeding duration with non-verbal intelligence in children of age 6 years. Maternal and paternal lifestyle, sociodemographic factors, child factors and maternal IQ were tested for their confounding effects on the association. Results We observed an initial association between breastfeeding duration and child IQ conferring an advantage of 0.32 (0.20 to 0.44) points for each additional month of breastfeeding. This association strongly attenuated to 0.09 (-0.03 to 0.21) points after adjustment for child factors, sociodemographic factors, parental lifestyle factors and maternal IQ. Similarly, the associations with breastfeeding duration as a categorical variable largely disappeared after confounding factors were added to the models. Conclusions The association between breastfeeding and child IQ can be largely explained by sociodemographic factors, parental lifestyle and maternal IQ. Our results cannot confirm beneficial effects of breastfeeding on child intelligence.