Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29 raters and an artificial intelligence-based software

Lennard Wolff*, Jiahang Su, on behalf of the MR CLEAN investigators, Derek Van Loon, Adriaan van Es, Pieter Jan van Doormaal, Charles Majoie, Wim van Zwam, Diederik Dippel, Aad van der Lugt, Theo van Walsum

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

6 Citations (Scopus)
26 Downloads (Pure)


Purpose: Outcome of endovascular treatment in acute ischemic stroke patients is depending on the collateral circulation maintaining blood flow to the ischemic territory. We evaluated the inter-rater reliability and accuracy of raters and an automated algorithm for assessing the collateral score (CS, range: 0–3) in acute ischemic stroke patients. Methods: Baseline CTA scans with an intracranial anterior occlusion from the MR CLEAN study (n=500) were used. For each core lab CS, ten CTA scans with sufficient quality were randomly selected. After a training session in collateral scoring, all selected CTA scans were individually evaluated for a visual CS by three groups: 7 radiologists, 13 junior and 9 senior radiology residents. Two additional radiologists scored CS to be used as reference, with a third providing a CS to produce a 2 out of 3 consensus CS in case of disagreement. An automated algorithm was also used to compute CS. Inter-rater agreement was reported with intraclass correlation coefficient (ICC). Accuracy of visual and automated CS were calculated. Results: 39 CTA scans were assessed (1 corrupt CTA-scan excluded). All groups showed a moderate ICC (0.689-0.780) in comparison to the reference standard. Overall human accuracy was 65± 7% and increased to 88± 5% for dichotomized CS (0–1, 2–3). Automated CS accuracy was 62%, and 90% for dichotomized CS. No significant difference in accuracy was found between groups with different levels of expertise. Conclusion: After training, inter-rater reliability in collateral scoring was not influenced by experience. Automated CS performs similar to residents and radiologists in determining a collateral score.

Original languageEnglish
Pages (from-to)2277-2284
Number of pages8
Issue number12
Early online date24 May 2022
Publication statusPublished - Dec 2022

Bibliographical note

Funding Information:
The MR CLEAN trial was funded in part by the Dutch Heart Foundation and by unrestricted grants from AngioCare BV, Medtronic/Covidien/EV3, MEDAC Gmbh/LAMEPRO, Penumbra Inc., Stryker, and Top Medical/Concentric. This study is executed within the CONTRAST consortium. The CONTRAST consortium is supported by Netherlands Cardiovascular Research Initiative, an initiative of the Dutch Heart Foundation, by the Brain Foundation Netherlands and powered by Health~Holland, Top Sector Life Sciences and receives unrestricted funding from Stryker, Penumbra, Medtronic, and Cerenovus.

Publisher Copyright:
© 2022, The Author(s).


Dive into the research topics of 'Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29 raters and an artificial intelligence-based software'. Together they form a unique fingerprint.

Cite this