Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation

Robin Camarasa; Daniel Bos; Jeroen Hendrikse; Paul Nederkoorn; Eline Kooi; Aad van der Lugt; Marleen de Bruijne

doi:10.1007/978-3-030-60365-6_4

Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation

Robin Camarasa^*, Daniel Bos, Jeroen Hendrikse, Paul Nederkoorn, Eline Kooi, Aad van der Lugt, Marleen de Bruijne

^*Corresponding author for this work

Research output: Chapter/Conference proceeding › Conference proceeding › Academic › peer-review

16 Citations (Scopus)

Abstract

Over the past decade, deep learning has become the gold standard for automatic medical image segmentation. Every segmentation task has an underlying uncertainty due to image resolution, annotation protocol, etc. Therefore, a number of methods and metrics have been proposed to quantify the uncertainty of neural networks mostly based on Bayesian deep learning, ensemble learning methods or output probability calibration. The aim of our research is to assess how reliable the different uncertainty metrics found in the literature are. We propose a quantitative and statistical comparison of uncertainty measures based on the relevance of the uncertainty map to predict misclassification. Four uncertainty metrics were compared over a set of 144 models. The application studied is the segmentation of the lumen and vessel wall of carotid arteries based on multiple sequences of magnetic resonance (MR) images in multi-center data.

Original language	English
Title of host publication	Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings
Editors	Carole H. Sudre, Hamid Fehri, Tal Arbel, Christian F. Baumgartner, Adrian Dalca, Ryutaro Tanno, Koen Van Leemput, William M. Wells, Aristeidis Sotiras, Bartlomiej Papiez, Enzo Ferrante, Sarah Parisot
Publisher	Springer Science+Business Media
Pages	32-41
Number of pages	10
ISBN (Print)	9783030603649
DOIs	https://doi.org/10.1007/978-3-030-60365-6_4
Publication status	Published - 2020
Event	2nd International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2020, and the 3rd International Workshop on Graphs in Biomedical Image Analysis, GRAIL 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020 - Lima, Peru Duration: 8 Oct 2020 → 8 Oct 2020

Publication series

Series	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12443 LNCS
ISSN	0302-9743

Conference

Conference	2nd International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2020, and the 3rd International Workshop on Graphs in Biomedical Image Analysis, GRAIL 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020
Country/Territory	Peru
City	Lima
Period	8/10/20 → 8/10/20

Bibliographical note

Funding Information:
Acknowledgments. This work was funded by Netherlands Organisation for Scientific Research (NWO) VICI project VI.C.182.042. The PARISK study was funded within the framework of CTMM, the Center for Translational Molecular Medicine, project PARISK (grant 01C-202), and supported by the Dutch Heart Foundation.

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

Access to Document

10.1007/978-3-030-60365-6_4

Cite this

Camarasa, R., Bos, D., Hendrikse, J., Nederkoorn, P., Kooi, E., van der Lugt, A., & de Bruijne, M. (2020). Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation. In C. H. Sudre, H. Fehri, T. Arbel, C. F. Baumgartner, A. Dalca, R. Tanno, K. Van Leemput, W. M. Wells, A. Sotiras, B. Papiez, E. Ferrante, & S. Parisot (Eds.), Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings (pp. 32-41). Springer Science+Business Media. https://doi.org/10.1007/978-3-030-60365-6_4

Camarasa, Robin ; Bos, Daniel ; Hendrikse, Jeroen et al. / Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings. editor / Carole H. Sudre ; Hamid Fehri ; Tal Arbel ; Christian F. Baumgartner ; Adrian Dalca ; Ryutaro Tanno ; Koen Van Leemput ; William M. Wells ; Aristeidis Sotiras ; Bartlomiej Papiez ; Enzo Ferrante ; Sarah Parisot. Springer Science+Business Media, 2020. pp. 32-41 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 12443 LNCS).

@inproceedings{b6bf050080644de2914d57950d04d7cb,

title = "Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation",

abstract = "Over the past decade, deep learning has become the gold standard for automatic medical image segmentation. Every segmentation task has an underlying uncertainty due to image resolution, annotation protocol, etc. Therefore, a number of methods and metrics have been proposed to quantify the uncertainty of neural networks mostly based on Bayesian deep learning, ensemble learning methods or output probability calibration. The aim of our research is to assess how reliable the different uncertainty metrics found in the literature are. We propose a quantitative and statistical comparison of uncertainty measures based on the relevance of the uncertainty map to predict misclassification. Four uncertainty metrics were compared over a set of 144 models. The application studied is the segmentation of the lumen and vessel wall of carotid arteries based on multiple sequences of magnetic resonance (MR) images in multi-center data.",

author = "Robin Camarasa and Daniel Bos and Jeroen Hendrikse and Paul Nederkoorn and Eline Kooi and {van der Lugt}, Aad and {de Bruijne}, Marleen",

note = "Funding Information: Acknowledgments. This work was funded by Netherlands Organisation for Scientific Research (NWO) VICI project VI.C.182.042. The PARISK study was funded within the framework of CTMM, the Center for Translational Molecular Medicine, project PARISK (grant 01C-202), and supported by the Dutch Heart Foundation. Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 2nd International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2020, and the 3rd International Workshop on Graphs in Biomedical Image Analysis, GRAIL 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020 ; Conference date: 08-10-2020 Through 08-10-2020",

year = "2020",

doi = "10.1007/978-3-030-60365-6_4",

language = "English",

isbn = "9783030603649",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science+Business Media",

pages = "32--41",

editor = "Sudre, {Carole H.} and Hamid Fehri and Tal Arbel and Baumgartner, {Christian F.} and Adrian Dalca and Ryutaro Tanno and {Van Leemput}, Koen and Wells, {William M.} and Aristeidis Sotiras and Bartlomiej Papiez and Enzo Ferrante and Sarah Parisot",

booktitle = "Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings",

address = "Germany",

}

Camarasa, R , Bos, D, Hendrikse, J, Nederkoorn, P, Kooi, E, van der Lugt, A & de Bruijne, M 2020, Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation. in CH Sudre, H Fehri, T Arbel, CF Baumgartner, A Dalca, R Tanno, K Van Leemput, WM Wells, A Sotiras, B Papiez, E Ferrante & S Parisot (eds), Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings. Springer Science+Business Media, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12443 LNCS, pp. 32-41, 2nd International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2020, and the 3rd International Workshop on Graphs in Biomedical Image Analysis, GRAIL 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020, Lima, Peru, 8/10/20. https://doi.org/10.1007/978-3-030-60365-6_4

Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation. / Camarasa, Robin ; Bos, Daniel; Hendrikse, Jeroen et al.
Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings. ed. / Carole H. Sudre; Hamid Fehri; Tal Arbel; Christian F. Baumgartner; Adrian Dalca; Ryutaro Tanno; Koen Van Leemput; William M. Wells; Aristeidis Sotiras; Bartlomiej Papiez; Enzo Ferrante; Sarah Parisot. Springer Science+Business Media, 2020. p. 32-41 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 12443 LNCS).

Research output: Chapter/Conference proceeding › Conference proceeding › Academic › peer-review

TY - GEN

T1 - Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation

AU - Camarasa, Robin

AU - Bos, Daniel

AU - Hendrikse, Jeroen

AU - Nederkoorn, Paul

AU - Kooi, Eline

AU - van der Lugt, Aad

AU - de Bruijne, Marleen

N1 - Funding Information: Acknowledgments. This work was funded by Netherlands Organisation for Scientific Research (NWO) VICI project VI.C.182.042. The PARISK study was funded within the framework of CTMM, the Center for Translational Molecular Medicine, project PARISK (grant 01C-202), and supported by the Dutch Heart Foundation. Publisher Copyright: © 2020, Springer Nature Switzerland AG.

PY - 2020

Y1 - 2020

N2 - Over the past decade, deep learning has become the gold standard for automatic medical image segmentation. Every segmentation task has an underlying uncertainty due to image resolution, annotation protocol, etc. Therefore, a number of methods and metrics have been proposed to quantify the uncertainty of neural networks mostly based on Bayesian deep learning, ensemble learning methods or output probability calibration. The aim of our research is to assess how reliable the different uncertainty metrics found in the literature are. We propose a quantitative and statistical comparison of uncertainty measures based on the relevance of the uncertainty map to predict misclassification. Four uncertainty metrics were compared over a set of 144 models. The application studied is the segmentation of the lumen and vessel wall of carotid arteries based on multiple sequences of magnetic resonance (MR) images in multi-center data.

AB - Over the past decade, deep learning has become the gold standard for automatic medical image segmentation. Every segmentation task has an underlying uncertainty due to image resolution, annotation protocol, etc. Therefore, a number of methods and metrics have been proposed to quantify the uncertainty of neural networks mostly based on Bayesian deep learning, ensemble learning methods or output probability calibration. The aim of our research is to assess how reliable the different uncertainty metrics found in the literature are. We propose a quantitative and statistical comparison of uncertainty measures based on the relevance of the uncertainty map to predict misclassification. Four uncertainty metrics were compared over a set of 144 models. The application studied is the segmentation of the lumen and vessel wall of carotid arteries based on multiple sequences of magnetic resonance (MR) images in multi-center data.

UR - http://www.scopus.com/inward/record.url?scp=85093121226&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-60365-6_4

DO - 10.1007/978-3-030-60365-6_4

M3 - Conference proceeding

AN - SCOPUS:85093121226

SN - 9783030603649

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 32

EP - 41

BT - Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings

A2 - Sudre, Carole H.

A2 - Fehri, Hamid

A2 - Arbel, Tal

A2 - Baumgartner, Christian F.

A2 - Dalca, Adrian

A2 - Tanno, Ryutaro

A2 - Van Leemput, Koen

A2 - Wells, William M.

A2 - Sotiras, Aristeidis

A2 - Papiez, Bartlomiej

A2 - Ferrante, Enzo

A2 - Parisot, Sarah

PB - Springer Science+Business Media

T2 - 2nd International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, UNSURE 2020, and the 3rd International Workshop on Graphs in Biomedical Image Analysis, GRAIL 2020, held in conjunction with the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020

Y2 - 8 October 2020 through 8 October 2020

ER -

Camarasa R , Bos D, Hendrikse J, Nederkoorn P, Kooi E, van der Lugt A et al. Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation. In Sudre CH, Fehri H, Arbel T, Baumgartner CF, Dalca A, Tanno R, Van Leemput K, Wells WM, Sotiras A, Papiez B, Ferrante E, Parisot S, editors, Uncertainty for Safe Utilization of Machine Learning in Medical Imaging, and Graphs in Biomedical Image Analysis - 2nd International Workshop, UNSURE 2020, and 3rd International Workshop, GRAIL 2020, Held in Conjunction with MICCAI 2020, Proceedings. Springer Science+Business Media. 2020. p. 32-41. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 12443 LNCS). doi: 10.1007/978-3-030-60365-6_4

Quantitative Comparison of Monte-Carlo Dropout Uncertainty Measures for Multi-class Segmentation

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this