Air pollution exposure during early-life is associated with altered brain development, but the precise periods of susceptibility are unknown. We aimed to investigate whether there are periods of susceptibility of air pollution between conception and preadolescence in relation to white matter microstructure and brain volumes at 9–12 years old. We used data of 3515 children from the Generation R Study, a population-based birth cohort from Rotterdam, the Netherlands (2002–2006). We estimated daily levels of nitrogen dioxide (NO2), and particulate matter (PM2.5 and PM2.5absorbance) at participants’ homes during pregnancy and childhood using land-use regression models. Diffusion tensor and structural brain images were obtained when children were 9–12 years of age, and we calculated fractional anisotropy and mean diffusivity, and several brain structure volumes. We performed distributed lag non-linear modeling adjusting for socioeconomic and lifestyle characteristics. We observed specific periods of susceptibility to all air pollutants from conception to age 5 years in association with lower fractional anisotropy and higher mean diffusivity that survived correction for multiple testing (e.g., −0.85 fractional anisotropy (95%CI -1.43; −0.27) per 5 μg/m3 increase in PM2.5 between conception and 4 years of age). We also observed certain periods of susceptibility to some air pollutants in relation to global brain and some subcortical brain volumes, but only the association between PM2.5 and putamen survived correction for multiple testing (172 mm3 (95%CI 57; 286) per 5 μg/m3 increase in PM2.5 between 4 months and 1.8 year of age). This study suggested that conception, pregnancy, infancy, toddlerhood, and early childhood seem to be susceptible periods to air pollution exposure for the development of white matter microstructure and the putamen volume. Longitudinal studies with repeated brain outcome measurements are needed for understanding the trajectories and the long-term effects of exposure to air pollution.