Cerebral small-vessel disease is thought to contribute to brain atrophy, but it remains unclear whether it affects the gray matter and white matter atrophy differentially. Retinal vessels provide a direct measure to study cerebral small-vessel disease in vivo. In a cohort of 1065 persons (mean age, 67.5 y and 51% women), from the population-based Rotterdam Study, we investigated how retinal vascular calibers relate to brain atrophy and to gray matter and white matter atrophy separately. Retinal arteriolar and venular calibers were semiautomatically measured on digitized fundus transparencies. Using automated quantification of MRI scans, we obtained whole-brain volume and volumes of gray matter and white matter. Both narrower arteriolar and wider venular calibers were associated with smaller brain volume, independent from each other. These associations were primarily driven by smaller white matter volume, whereas no associations were seen for gray matter volume. Adjustments for cardiovascular risk factors attenuated the results, but wider venular caliber remained borderline significantly associated with smaller white matter volume. Our data provide evidence that cerebral small-vessel disease contributes to brain atrophy primarily by affecting the cerebral white matter.