Interrater agreement in classifying infections during extracorporeal membrane oxygenation

Karlijn Verkerk*, Lara C.A. Pladet, Christiaan L. Meuwese, Dirk W. Donker, Lennie P.G. Derde, Olaf L. Cremer

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

1 Citation (Web of Science)
16 Downloads (Pure)


Infectious complications are common during extracorporeal membrane oxygenation (ECMO) and may negatively impact outcomes. However, there is considerable variation in the reported rates of incidence, which hampers the use of infections as a quality benchmark for ECMO centers. To assess the contributing role of poor interrater agreement, three independent raters reviewed medical records from all intensive care unit (ICU) patients who received ECMO for >24 h in our tertiary center between October 2019 and October 2021 for suspected episodes of infection, which were rated based on their date of onset and presumed site/diagnosis. To establish a gold standard, any discrepancies were resolved using an expert panel consisting of two intensivists/infectious disease specialists. During 83 ECMO-runs in 77 patients, we observed a total of 62 adjudicated infectious episodes (incidence rate 62, 95% CI: 48–80, per 1000 days at risk). Among 81 episodes suspected by at least one observer, 66 (81%) were identified by two, and only 44 (54%) by all three raters, resulting in Fleiss’ kappa of 0.10 (95% CI: 0.00–0.19; slight agreement). However, if raters concurred regarding infection onset, subsequent agreement on infection site was good (concordance 89%; kappa 0.85, 95% CI: 0.72–0.98; near perfect agreement). In conclusion, adjudication of infectious episodes during ECMO is associated with poor interrater agreement regarding occurrence—but not site—of infection. This finding might partially explain the significant disparities observed in reported infection rates during ECMO, emphasizing the need for caution when interpreting infection data in this particular population due to the potential for inherent measurement error.

Original languageEnglish
Pages (from-to)597-601
Number of pages5
JournalInternational Journal of Artificial Organs
Issue number10-11
Early online date19 Aug 2023
Publication statusPublished - 1 Oct 2023

Bibliographical note

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The primary source of funding of this manuscript is Stichting Gezondheidszorg Spaarneland (SGS) Fund, Zilveren Kruis Achmea.

Publisher Copyright: © The Author(s) 2023.


Dive into the research topics of 'Interrater agreement in classifying infections during extracorporeal membrane oxygenation'. Together they form a unique fingerprint.

Cite this