Abstract
Machine learning methods exploiting multi-parametric biomarkers, especially based on neuroimaging, have huge potential to improve early diagnosis of dementia and to predict which individuals are at-risk of developing dementia. To benchmark algorithms in the field of machine learning and neuroimaging in dementia and assess their potential for use in clinical practice and clinical trials, seven grand challenges have been organized in the last decade: MIRIAD (2012), Alzheimer's Disease Big Data DREAM (2014), CADDementia (2014), Machine Learning Challenge (2014), MCI Neuroimaging (2017), TADPOLE (2017), and the Predictive Analytics Competition (2019). Based on two challenge evaluation frameworks, we analyzed how these grand challenges are complementing each other regarding research questions, datasets, validation approaches, results and impact. The seven grand challenges addressed questions related to screening, clinical status estimation, prediction and monitoring in (pre-clinical) dementia. There was little overlap in clinical questions, tasks and performance metrics. Whereas this aids providing insight on a broad range of questions, it also limits the validation of results across challenges. The validation process itself was mostly comparable between challenges, using similar methods for ensuring objective comparison, uncertainty estimation and statistical testing. In general, winning algorithms performed rigorous data pre-processing and combined a wide range of input features. Despite high state-of-the-art performances, most of the methods evaluated by the challenges are not clinically used. To increase impact, future challenges could pay more attention to statistical analysis of which factors (i.e., features, models) relate to higher performance, to clinical questions beyond Alzheimer's disease, and to using testing data beyond the Alzheimer's Disease Neuroimaging Initiative. Grand challenges would be an ideal venue for assessing the generalizability of algorithm performance to unseen data of other cohorts. Key for increasing impact in this way are larger testing data sizes, which could be reached by sharing algorithms rather than data to exploit data that cannot be shared. Given the potential and lessons learned in the past ten years, we are excited by the prospects of grand challenges in machine learning and neuroimaging for the next ten years and beyond.
| Original language | English |
|---|---|
| Article number | 119083 |
| Journal | NeuroImage |
| Volume | 253 |
| DOIs | |
| Publication status | Published - Jun 2022 |
Bibliographical note
Funding Information:E.E. Bron acknowledges support from Medical Delta (Diagnostics 3.0: Dementia and Stroke), Dutch Heart Foundation (PPP Allowance, 2018B011), the Dutch CardioVascular Alliance (Heart-Brain Connection: CVON2012-06, CVON2018-28) and the Netherlands eScience Center (2018 Young eScientist award). N.P. Oxtoby is a UKRI Future Leaders Fellow (MR/S03546X/1). N.P. Oxtoby and D.C. Alexander acknowledge funding from the E-DADS project (EU JPND), and the National Institute for Health Research University College London Hospitals Biomedical Research Centre. This work is part of the EuroPOND initiative, which is funded by the European Union ’s Horizon 2020 research and innovation program under grant agreement no. 666992 . Part of this work was funded by the Helmholtz Imaging Platform (HIP), a platform of the Helmholtz Incubator on Information and Data Science. We further acknowledge the Deep Dementia Phenotyping (DEMON) network, 13 13 an international network for the application of data science and AI to dementia research..
Publisher Copyright:
© 2022 The Author(s)
