Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Todd J. Levy; Kevin Coppa; Jinxuan Cang; Douglas P. Barnaby; Marc D. Paradis; Stuart L. Cohen; Alex Makhnevich; David van Klaveren; David M. Kent; Karina W. Davidson; Jamie S. Hirsch; Theodoros P. Zanos

doi:10.1038/s41467-022-34646-2

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Todd J. Levy, Kevin Coppa, Jinxuan Cang, Douglas P. Barnaby, Marc D. Paradis, Stuart L. Cohen, Alex Makhnevich, David van Klaveren, David M. Kent, Karina W. Davidson, Jamie S. Hirsch, Theodoros P. Zanos^*

^*Corresponding author for this work

Public Health

Research output: Contribution to journal › Article › Academic › peer-review

10 Citations (Scopus)

16 Downloads (Pure)

Abstract

Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

Original language	English
Article number	6812
Journal	Nature Communications
Volume	13
Issue number	1
DOIs	https://doi.org/10.1038/s41467-022-34646-2
Publication status	Published - 10 Nov 2022

Bibliographical note

Acknowledgements
We acknowledge and honor all our Northwell team members who consistently put themselves in harm’s way during the COVID-19 pandemic. Their vital contribution to knowledge about COVID-19 and sacrifices on the behalf of patients made this possible. We would also like to acknowledge Challace Pahlevan-Ibrekic and Jackson Yeh for regulatory guidance and help with data de-identification and sharing. This work was supported by grants R24AG064191 from the National Institute on Aging, R01LM012836 from the U.S. National Library of Medicine, K23HL145114 from the National Heart, Lung, and Blood Institute (KWD), and ME-1606-35555 from the Patient-Centered Outcomes Research Institute (PCORI) Award (DvK, DMK, TPZ).

Publisher Copyright: © 2022, The Author(s).

Access to Document

10.1038/s41467-022-34646-2Licence: CC BY

s41467-022-34646-2Final published version, 5.9 MBLicence: CC BY

Cite this

Levy, T. J., Coppa, K., Cang, J., Barnaby, D. P., Paradis, M. D., Cohen, S. L., Makhnevich, A., van Klaveren, D., Kent, D. M., Davidson, K. W., Hirsch, J. S., & Zanos, T. P. (2022). Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients. Nature Communications, 13(1), Article 6812. https://doi.org/10.1038/s41467-022-34646-2

@article{05bcafa8f6ad4399b73f2340855d62fb,

title = "Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients",

abstract = "Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.",

author = "Levy, {Todd J.} and Kevin Coppa and Jinxuan Cang and Barnaby, {Douglas P.} and Paradis, {Marc D.} and Cohen, {Stuart L.} and Alex Makhnevich and {van Klaveren}, David and Kent, {David M.} and Davidson, {Karina W.} and Hirsch, {Jamie S.} and Zanos, {Theodoros P.}",

note = "Acknowledgements We acknowledge and honor all our Northwell team members who consistently put themselves in harm{\textquoteright}s way during the COVID-19 pandemic. Their vital contribution to knowledge about COVID-19 and sacrifices on the behalf of patients made this possible. We would also like to acknowledge Challace Pahlevan-Ibrekic and Jackson Yeh for regulatory guidance and help with data de-identification and sharing. This work was supported by grants R24AG064191 from the National Institute on Aging, R01LM012836 from the U.S. National Library of Medicine, K23HL145114 from the National Heart, Lung, and Blood Institute (KWD), and ME-1606-35555 from the Patient-Centered Outcomes Research Institute (PCORI) Award (DvK, DMK, TPZ). Publisher Copyright: {\textcopyright} 2022, The Author(s).",

year = "2022",

month = nov,

day = "10",

doi = "10.1038/s41467-022-34646-2",

language = "English",

volume = "13",

journal = "Nature Communications",

issn = "2041-1723",

publisher = "Nature Research",

number = "1",

}

TY - JOUR

T1 - Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

AU - Levy, Todd J.

AU - Coppa, Kevin

AU - Cang, Jinxuan

AU - Barnaby, Douglas P.

AU - Paradis, Marc D.

AU - Cohen, Stuart L.

AU - Makhnevich, Alex

AU - van Klaveren, David

AU - Kent, David M.

AU - Davidson, Karina W.

AU - Hirsch, Jamie S.

AU - Zanos, Theodoros P.

N1 - Acknowledgements We acknowledge and honor all our Northwell team members who consistently put themselves in harm’s way during the COVID-19 pandemic. Their vital contribution to knowledge about COVID-19 and sacrifices on the behalf of patients made this possible. We would also like to acknowledge Challace Pahlevan-Ibrekic and Jackson Yeh for regulatory guidance and help with data de-identification and sharing. This work was supported by grants R24AG064191 from the National Institute on Aging, R01LM012836 from the U.S. National Library of Medicine, K23HL145114 from the National Heart, Lung, and Blood Institute (KWD), and ME-1606-35555 from the Patient-Centered Outcomes Research Institute (PCORI) Award (DvK, DMK, TPZ). Publisher Copyright: © 2022, The Author(s).

PY - 2022/11/10

Y1 - 2022/11/10

N2 - Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

AB - Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

UR - http://www.scopus.com/inward/record.url?scp=85141549053&partnerID=8YFLogxK

U2 - 10.1038/s41467-022-34646-2

DO - 10.1038/s41467-022-34646-2

M3 - Article

C2 - 36357420

AN - SCOPUS:85141549053

SN - 2041-1723

VL - 13

JO - Nature Communications

JF - Nature Communications

IS - 1

M1 - 6812

ER -

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this