Interpreting Incremental Value of Markers Added to Risk Prediction Models

MJ Pencina, RB D'Agostino, KM Pencina, Cecile Janssens, P Greenland

Research output: Contribution to journalArticleAcademicpeer-review

308 Citations (Scopus)

Abstract

The discrimination of a risk prediction model measures that models ability to distinguish between subjects with and without events. The area under the receiver operating characteristic curve (AUC) is a popular measure of discrimination. However, the AUC has recently been criticized for its insensitivity in model comparisons in which the baseline model has performed well. Thus, 2 other measures have been proposed to capture improvement in discrimination for nested models: the integrated discrimination improvement and the continuous net reclassification improvement. In the present study, the authors use mathematical relations and numerical simulations to quantify the improvement in discrimination offered by candidate markers of different strengths as measured by their effect sizes. They demonstrate that the increase in the AUC depends on the strength of the baseline model, which is true to a lesser degree for the integrated discrimination improvement. On the other hand, the continuous net reclassification improvement depends only on the effect size of the candidate variable and its correlation with other predictors. These measures are illustrated using the Framingham model for incident atrial fibrillation. The authors conclude that the increase in the AUC, integrated discrimination improvement, and net reclassification improvement offer complementary information and thus recommend reporting all 3 alongside measures characterizing the performance of the final model.
Original languageUndefined/Unknown
Pages (from-to)473-481
Number of pages9
JournalAmerican Journal of Epidemiology
Volume176
Issue number6
DOIs
Publication statusPublished - 2012

Cite this