Early experiences with crowdsourcing airway annotations in chest CT

Veronika Cheplygina; Adria Perez-Rovira; Wieying Kuo; Harm A.W.M. Tiddens; Marleen de Bruijne

doi:10.1007/978-3-319-46976-8_22

Early experiences with crowdsourcing airway annotations in chest CT

Veronika Cheplygina^*, Adria Perez-Rovira, Wieying Kuo, Harm A.W.M. Tiddens, Marleen de Bruijne

^*Corresponding author for this work

Research output: Chapter/Conference proceeding › Conference proceeding › Academic › peer-review

16 Citations (Scopus)

Abstract

Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

Original language	English
Title of host publication	Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings
Editors	Zhi Lu, Vasileios Belagiannis, Joao Manuel R.S. Tavares, Jaime S. Cardoso, Andrew Bradley, Joao Paulo Papa, Jacinto C. Nascimento, Marco Loog, Julien Cornebise, Gustavo Carneiro, Diana Mateus, Loic Peter
Pages	209-218
Number of pages	10
DOIs	https://doi.org/10.1007/978-3-319-46976-8_22
Publication status	Published - 2016
Event	1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2016 and 2nd International Workshop on Deep Learning in Medical Image Analysis, DLMIA 2016 held in conjunction with 19th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2016 - Athens, Greece Duration: 21 Oct 2016 → 21 Oct 2016

Publication series

Series	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10008 LNCS
ISSN	0302-9743

Conference

Conference	1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2016 and 2nd International Workshop on Deep Learning in Medical Image Analysis, DLMIA 2016 held in conjunction with 19th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2016
Country/Territory	Greece
City	Athens
Period	21/10/16 → 21/10/16

Bibliographical note

Funding Information:
This research was partially funded by the research project “Transfer learning in biomedical image analysis” which is financed by the Netherlands Organization for Scientific Research (NWO) grant no. 639.022.010. We gratefully acknowledge Dr. Daniel Kondermann of Heidelberg University for his help with crowdsourcing, and the anonymous reviewers for their constructive comments.

Publisher Copyright:
© Springer International Publishing AG 2016.

Access to Document

10.1007/978-3-319-46976-8_22

Cite this

Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H. A. W. M., & de Bruijne, M. (2016). Early experiences with crowdsourcing airway annotations in chest CT. In Z. Lu, V. Belagiannis, J. M. R. S. Tavares, J. S. Cardoso, A. Bradley, J. P. Papa, J. C. Nascimento, M. Loog, J. Cornebise, G. Carneiro, D. Mateus, & L. Peter (Eds.), Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings (pp. 209-218) https://doi.org/10.1007/978-3-319-46976-8_22

Cheplygina, Veronika ; Perez-Rovira, Adria ; Kuo, Wieying et al. / Early experiences with crowdsourcing airway annotations in chest CT. Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings. editor / Zhi Lu ; Vasileios Belagiannis ; Joao Manuel R.S. Tavares ; Jaime S. Cardoso ; Andrew Bradley ; Joao Paulo Papa ; Jacinto C. Nascimento ; Marco Loog ; Julien Cornebise ; Gustavo Carneiro ; Diana Mateus ; Loic Peter. 2016. pp. 209-218 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 10008 LNCS).

@inproceedings{847d4b49a8fd47619721182609af5b41,

title = "Early experiences with crowdsourcing airway annotations in chest CT",

abstract = "Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.",

author = "Veronika Cheplygina and Adria Perez-Rovira and Wieying Kuo and Tiddens, {Harm A.W.M.} and {de Bruijne}, Marleen",

note = "Funding Information: This research was partially funded by the research project “Transfer learning in biomedical image analysis” which is financed by the Netherlands Organization for Scientific Research (NWO) grant no. 639.022.010. We gratefully acknowledge Dr. Daniel Kondermann of Heidelberg University for his help with crowdsourcing, and the anonymous reviewers for their constructive comments. Publisher Copyright: {\textcopyright} Springer International Publishing AG 2016.; 1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2016 and 2nd International Workshop on Deep Learning in Medical Image Analysis, DLMIA 2016 held in conjunction with 19th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2016 ; Conference date: 21-10-2016 Through 21-10-2016",

year = "2016",

doi = "10.1007/978-3-319-46976-8_22",

language = "English",

isbn = "9783319469751",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science+Business Media",

pages = "209--218",

editor = "Zhi Lu and Vasileios Belagiannis and Tavares, {Joao Manuel R.S.} and Cardoso, {Jaime S.} and Andrew Bradley and Papa, {Joao Paulo} and Nascimento, {Jacinto C.} and Marco Loog and Julien Cornebise and Gustavo Carneiro and Diana Mateus and Loic Peter",

booktitle = "Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings",

}

Cheplygina, V, Perez-Rovira, A, Kuo, W, Tiddens, HAWM & de Bruijne, M 2016, Early experiences with crowdsourcing airway annotations in chest CT. in Z Lu, V Belagiannis, JMRS Tavares, JS Cardoso, A Bradley, JP Papa, JC Nascimento, M Loog, J Cornebise, G Carneiro, D Mateus & L Peter (eds), Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10008 LNCS, pp. 209-218, 1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2016 and 2nd International Workshop on Deep Learning in Medical Image Analysis, DLMIA 2016 held in conjunction with 19th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2016, Athens, Greece, 21/10/16. https://doi.org/10.1007/978-3-319-46976-8_22

Early experiences with crowdsourcing airway annotations in chest CT. / Cheplygina, Veronika; Perez-Rovira, Adria; Kuo, Wieying et al.
Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings. ed. / Zhi Lu; Vasileios Belagiannis; Joao Manuel R.S. Tavares; Jaime S. Cardoso; Andrew Bradley; Joao Paulo Papa; Jacinto C. Nascimento; Marco Loog; Julien Cornebise; Gustavo Carneiro; Diana Mateus; Loic Peter. 2016. p. 209-218 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 10008 LNCS).

Research output: Chapter/Conference proceeding › Conference proceeding › Academic › peer-review

TY - GEN

T1 - Early experiences with crowdsourcing airway annotations in chest CT

AU - Cheplygina, Veronika

AU - Perez-Rovira, Adria

AU - Kuo, Wieying

AU - Tiddens, Harm A.W.M.

AU - de Bruijne, Marleen

N1 - Funding Information: This research was partially funded by the research project “Transfer learning in biomedical image analysis” which is financed by the Netherlands Organization for Scientific Research (NWO) grant no. 639.022.010. We gratefully acknowledge Dr. Daniel Kondermann of Heidelberg University for his help with crowdsourcing, and the anonymous reviewers for their constructive comments. Publisher Copyright: © Springer International Publishing AG 2016.

PY - 2016

Y1 - 2016

N2 - Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

AB - Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

UR - http://www.scopus.com/inward/record.url?scp=84992489314&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-46976-8_22

DO - 10.1007/978-3-319-46976-8_22

M3 - Conference proceeding

AN - SCOPUS:84992489314

SN - 9783319469751

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 209

EP - 218

BT - Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings

A2 - Lu, Zhi

A2 - Belagiannis, Vasileios

A2 - Tavares, Joao Manuel R.S.

A2 - Cardoso, Jaime S.

A2 - Bradley, Andrew

A2 - Papa, Joao Paulo

A2 - Nascimento, Jacinto C.

A2 - Loog, Marco

A2 - Cornebise, Julien

A2 - Carneiro, Gustavo

A2 - Mateus, Diana

A2 - Peter, Loic

T2 - 1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2016 and 2nd International Workshop on Deep Learning in Medical Image Analysis, DLMIA 2016 held in conjunction with 19th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2016

Y2 - 21 October 2016 through 21 October 2016

ER -

Cheplygina V, Perez-Rovira A, Kuo W, Tiddens HAWM, de Bruijne M. Early experiences with crowdsourcing airway annotations in chest CT. In Lu Z, Belagiannis V, Tavares JMRS, Cardoso JS, Bradley A, Papa JP, Nascimento JC, Loog M, Cornebise J, Carneiro G, Mateus D, Peter L, editors, Deep Learning and Data Labeling for Medical Applications - 1st International Workshop, LABELS 2016, and 2nd International Workshop, DLMIA 2016 Held in Conjunction with MICCAI 2016, Proceedings. 2016. p. 209-218. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 10008 LNCS). doi: 10.1007/978-3-319-46976-8_22

Early experiences with crowdsourcing airway annotations in chest CT

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this