There is consistent evidence that brain volume changes in early and late life. Most longitudinal studies usually only span a few years and include a limited number of participants. In this review, we integrate findings from 56 longitudinal magnetic resonance imaging (MRI) studies on whole brain volume change in healthy individuals. The individual longitudinal MRI studies describe only the development in a limited age range. In total, 2,211 participants were included. Age at first measurement varied between 4 and 88 years of age. The studies included in this review were performed using a large range of methods (e.g., different scanner protocols and different acquisition parameters). We applied a weighted regression analysis to estimate the age dependency of the rate of relative annual brain volume change across studies. The results indicate that whole brain volume changes throughout the life span. A wave of growth occurs during childhood/adolescence, where around 9 years of age a 1% annual brain growth is found which levels off until at age 13 a gradual volume decrease sets in. During young adulthood, between ~18 and 35 years of age, possibly another wave of growth occurs or at least a period of no brain tissue loss. After age 35 years, a steady volume loss is found of 0.2% per year, which accelerates gradually to an annual brain volume loss of 0.5% at age 60. The brains of people over 60 years of age show a steady volume loss of more than 0.5%. Understanding the mechanisms underlying these plastic brain changes may contribute to distinguishing progressive brain changes in psychiatric and neurological diseases from healthy aging processes.