External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature

Benjamin S. Wessler*, Jason Nelson, Jinny G. Park, Hannah McGinnes, Gaurav Gulati, Riley Brazil, Ben Van Calster, David Van Klaveren, Esmee Venema, Ewout Steyerberg, Jessica K. Paulus, David M. Kent

*Corresponding author for this work

Research output: Contribution to journalReview articleAcademicpeer-review

30 Citations (Scopus)
27 Downloads (Pure)

Abstract

Background: There are many clinical prediction models (CPMs) available to inform treatment decisions for patients with cardiovascular disease. However, the extent to which they have been externally tested, and how well they generally perform has not been broadly evaluated. Methods: A SCOPUS citation search was run on March 22, 2017 to identify external validations of cardiovascular CPMs in the Tufts Predictive Analytics and Comparative Effectiveness CPM Registry. We assessed the extent of external validation, performance heterogeneity across databases, and explored factors associated with model performance, including a global assessment of the clinical relatedness between the derivation and validation data. Results: We identified 2030 external validations of 1382 CPMs. Eight hundred seven (58%) of the CPMs in the Registry have never been externally validated. On average, there were 1.5 validations per CPM (range, 0-94). The median external validation area under the receiver operating characteristic curve was 0.73 (25th-75th percentile [interquartile range (IQR)], 0.66-0.79), representing a median percent decrease in discrimination of -11.1% (IQR, -32.4% to +2.7%) compared with performance on derivation data. 81% (n=1333) of validations reporting area under the receiver operating characteristic curve showed discrimination below that reported in the derivation dataset. 53% (n=983) of the validations report some measure of CPM calibration. For CPMs evaluated more than once, there was typically a large range of performance. Of 1702 validations classified by relatedness, the percent change in discrimination was -3.7% (IQR, -13.2 to 3.1) for closely related validations (n=123), -9.0 (IQR, -27.6 to 3.9) for related validations (n=862), and -17.2% (IQR, -42.3 to 0) for distantly related validations (n=717; P<0.001). Conclusions: Many published cardiovascular CPMs have never been externally validated, and for those that have, apparent performance during development is often overly optimistic. A single external validation appears insufficient to broadly understand the performance heterogeneity across different settings.

Original languageEnglish
Article numberE007858
JournalCirculation: Cardiovascular Quality and Outcomes
Volume14
Issue number8
DOIs
Publication statusPublished - 3 Aug 2021

Bibliographical note

Sources of Funding:
Research reported in this work was funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (ME-1606-35555). The views, statements, opinions presented in this work are solely the responsibility of the author(s) and do not necessarily represent the views of the PCORI, its Board of Governors, or Methodology Committee. Dr Wessler is supported by K23AG055667 from National Institutes of Health (NIH)–National Institute on Aging (NIA) and R03AG056447 from NIH-NIA.

Publisher Copyright: © 2021 Lippincott Williams and Wilkins. All rights reserved.

Fingerprint

Dive into the research topics of 'External Validations of Cardiovascular Clinical Prediction Models: A Large-Scale Review of the Literature'. Together they form a unique fingerprint.

Cite this