Background: Previous studies identifying hearing loss as a promising modifiable risk factor for cognitive decline mostly adjusted for baseline age solely. As such a faster cognitive decline at a higher age, which is expected considering the non-linear relationship between cognition and age, may have been overlooked. Therefore it remains uncertain whether effects of hearing loss on cognitive decline extend beyond age-related declines of cognitive function. Methods: 3,590 non-demented participants were eligible for analysis at baseline, and a maximum of 837 participants were eligible for the longitudinal analysis. Hearing loss was defined at baseline. Cognitive function was measured at baseline and at follow-up (4.4 years [SD: 0.2]). Multivariable linear regression analysis was used for the cross-sectional analysis. Linear mixed models were used to assess the longitudinal association between hearing loss and cognitive decline over time while adjusting for confounders and the interaction of age and follow-up time. Results: Hearing loss was associated with lower cognitive function at baseline. Moreover, hearing loss was associated with accelerated cognitive decline over time on a memory test. After additionally adjusting for the interaction between age and follow-up time, we found that hearing loss did not accelerate cognitive decline anymore. Conclusions: Hearing loss was associated with lower cognitive function at baseline and accelerated cognitive decline on a memory test. The association between hearing loss and accelerated cognitive decline was non-significant after additional adjustment for non-linear age effects. More evidence is needed to ensure the role of hearing loss as a modifiable risk factor for cognitive decline.