Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis

William Rojas-Carabali; Carlos Cifuentes-González; Xin Wei; Ikhwanuliman Putera; Alok Sen; Zheng Xian Thng; Rupesh Agrawal; Tobias Elze; Lucia Sobrin; John H. Kempen; Bernett Lee; Jyotirmay Biswas; Quan Dong Nguyen; Vishali Gupta; Alejandra de-la-Torre; Rupesh Agrawal

doi:10.1080/09273948.2023.2253471

Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis

William Rojas-Carabali
, Carlos Cifuentes-González
, Xin Wei
, Ikhwanuliman Putera
, Alok Sen
, Zheng Xian Thng
, Rupesh Agrawal^*
, Tobias Elze
, Lucia Sobrin
, John H. Kempen
, Bernett Lee
, Jyotirmay Biswas
, Quan Dong Nguyen
, Vishali Gupta
, Alejandra de-la-Torre
, Rupesh Agrawal^*

^*Corresponding author for this work

Tan Tock Seng Hospital
Lee Kong Chian School of Medicine
Universidad del Rosario
Sadguru Netra Chikatsalya
Harvard University
Community Ophthalmology
Addis Ababa University
Myungsung Christian Medical Center
Medical Research Foundation, Chennai
Stanford University
Postgraduate Institute of Medical Education and Research
Duke-NUS Graduate Medical School
Moorfields Eye Hospital NHS Foundation Trust
Singapore Eye Research Institute
University of Indonesia

Research output: Contribution to journal › Article › Academic › peer-review

44 Citations (Scopus)

71 Downloads (Pure)

Abstract

Introduction:

Accurate diagnosis and timely management are vital for favorable uveitis outcomes. Artificial Intelligence (AI) holds promise in medical decision-making, particularly in ophthalmology. Yet, the diagnostic precision and management advice from AI-based uveitis chatbots lack assessment.

Methods:

We appraised diagnostic accuracy and management suggestions of an AI-based chatbot, ChatGPT, versus five uveitis-trained ophthalmologists, using 25 standard cases aligned with new Uveitis Nomenclature guidelines. Participants predicted likely diagnoses, two differentials, and next management steps. Comparative success rates were computed.

Results:

Ophthalmologists excelled (60–92%) in likely diagnosis, exceeding AI (60%). Considering fully and partially accurate diagnoses, ophthalmologists achieved 76–100% success; AI attained 72%. Despite an 8% AI improvement, its overall performance lagged. Ophthalmologists and AI agreed on diagnosis in 48% cases, with 91.6% exhibiting concurrence in management plans.

Conclusions:

The study underscores AI chatbots' potential in uveitis diagnosis and management, indicating their value in reducing diagnostic errors. Further research is essential to enhance AI chatbot precision in diagnosis and recommendations.

Original language	English
Pages (from-to)	1526-1531
Number of pages	6
Journal	Ocular Immunology and Inflammation
Volume	32
Issue number	8
DOIs	https://doi.org/10.1080/09273948.2023.2253471
Publication status	Published - 2024

Bibliographical note

Funding Information:
Rupesh Agrawal receives clinician-scientist funding from the National Medical Research Council (NMRC, Singapore, grant number: CSAINV19nov-0007). The project was supported by National Medical Research Council Grant by Ministry of Health, Singapore. Grant number: NMRC/MOH-CNIG19may-0001, Grant title: Optical coherence tomography-based novel outcome measures for disease monitoring in intraocular inflammatory and infectious diseases. Study PI: Dr Wei Xin and study mentor: Prof Rupesh Agrawal.

Publisher Copyright:
© 2023 Taylor & Francis Group, LLC.

Access to Document

10.1080/09273948.2023.2253471

Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in UveitisFinal published version, 3.18 MBLicence: Article 25fa Dutch Copyright Act

Cite this

Rojas-Carabali, W., Cifuentes-González, C., Wei, X., Putera, I., Sen, A., Thng, Z. X., Agrawal, R., Elze, T., Sobrin, L., Kempen, J. H., Lee, B., Biswas, J., Nguyen, Q. D., Gupta, V., de-la-Torre, A., & Agrawal, R. (2024). Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis. Ocular Immunology and Inflammation, 32(8), 1526-1531. https://doi.org/10.1080/09273948.2023.2253471

@article{d3f533b23f63494f821575356f61823d,

title = "Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis",

abstract = "Introduction: Accurate diagnosis and timely management are vital for favorable uveitis outcomes. Artificial Intelligence (AI) holds promise in medical decision-making, particularly in ophthalmology. Yet, the diagnostic precision and management advice from AI-based uveitis chatbots lack assessment. Methods: We appraised diagnostic accuracy and management suggestions of an AI-based chatbot, ChatGPT, versus five uveitis-trained ophthalmologists, using 25 standard cases aligned with new Uveitis Nomenclature guidelines. Participants predicted likely diagnoses, two differentials, and next management steps. Comparative success rates were computed.Results: Ophthalmologists excelled (60{\textendash}92\%) in likely diagnosis, exceeding AI (60\%). Considering fully and partially accurate diagnoses, ophthalmologists achieved 76{\textendash}100\% success; AI attained 72\%. Despite an 8\% AI improvement, its overall performance lagged. Ophthalmologists and AI agreed on diagnosis in 48\% cases, with 91.6\% exhibiting concurrence in management plans. Conclusions: The study underscores AI chatbots' potential in uveitis diagnosis and management, indicating their value in reducing diagnostic errors. Further research is essential to enhance AI chatbot precision in diagnosis and recommendations.",

author = "William Rojas-Carabali and Carlos Cifuentes-Gonz{\'a}lez and Xin Wei and Ikhwanuliman Putera and Alok Sen and Thng, \{Zheng Xian\} and Rupesh Agrawal and Tobias Elze and Lucia Sobrin and Kempen, \{John H.\} and Bernett Lee and Jyotirmay Biswas and Nguyen, \{Quan Dong\} and Vishali Gupta and Alejandra de-la-Torre and Rupesh Agrawal",

note = "Funding Information: Rupesh Agrawal receives clinician-scientist funding from the National Medical Research Council (NMRC, Singapore, grant number: CSAINV19nov-0007). The project was supported by National Medical Research Council Grant by Ministry of Health, Singapore. Grant number: NMRC/MOH-CNIG19may-0001, Grant title: Optical coherence tomography-based novel outcome measures for disease monitoring in intraocular inflammatory and infectious diseases. Study PI: Dr Wei Xin and study mentor: Prof Rupesh Agrawal. Publisher Copyright: {\textcopyright} 2023 Taylor \& Francis Group, LLC.",

year = "2024",

doi = "10.1080/09273948.2023.2253471",

language = "English",

volume = "32",

pages = "1526--1531",

journal = "Ocular Immunology and Inflammation",

issn = "0927-3948",

publisher = "Taylor \& Francis Ltd (UK)",

number = "8",

}

Rojas-Carabali, W, Cifuentes-González, C, Wei, X, Putera, I, Sen, A, Thng, ZX, Agrawal, R, Elze, T, Sobrin, L, Kempen, JH, Lee, B, Biswas, J, Nguyen, QD, Gupta, V, de-la-Torre, A & Agrawal, R 2024, 'Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis', Ocular Immunology and Inflammation, vol. 32, no. 8, pp. 1526-1531. https://doi.org/10.1080/09273948.2023.2253471

TY - JOUR

T1 - Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis

AU - Rojas-Carabali, William

AU - Cifuentes-González, Carlos

AU - Wei, Xin

AU - Putera, Ikhwanuliman

AU - Sen, Alok

AU - Thng, Zheng Xian

AU - Agrawal, Rupesh

AU - Elze, Tobias

AU - Sobrin, Lucia

AU - Kempen, John H.

AU - Lee, Bernett

AU - Biswas, Jyotirmay

AU - Nguyen, Quan Dong

AU - Gupta, Vishali

AU - de-la-Torre, Alejandra

AU - Agrawal, Rupesh

N1 - Funding Information: Rupesh Agrawal receives clinician-scientist funding from the National Medical Research Council (NMRC, Singapore, grant number: CSAINV19nov-0007). The project was supported by National Medical Research Council Grant by Ministry of Health, Singapore. Grant number: NMRC/MOH-CNIG19may-0001, Grant title: Optical coherence tomography-based novel outcome measures for disease monitoring in intraocular inflammatory and infectious diseases. Study PI: Dr Wei Xin and study mentor: Prof Rupesh Agrawal. Publisher Copyright: © 2023 Taylor & Francis Group, LLC.

PY - 2024

Y1 - 2024

N2 - Introduction: Accurate diagnosis and timely management are vital for favorable uveitis outcomes. Artificial Intelligence (AI) holds promise in medical decision-making, particularly in ophthalmology. Yet, the diagnostic precision and management advice from AI-based uveitis chatbots lack assessment. Methods: We appraised diagnostic accuracy and management suggestions of an AI-based chatbot, ChatGPT, versus five uveitis-trained ophthalmologists, using 25 standard cases aligned with new Uveitis Nomenclature guidelines. Participants predicted likely diagnoses, two differentials, and next management steps. Comparative success rates were computed.Results: Ophthalmologists excelled (60–92%) in likely diagnosis, exceeding AI (60%). Considering fully and partially accurate diagnoses, ophthalmologists achieved 76–100% success; AI attained 72%. Despite an 8% AI improvement, its overall performance lagged. Ophthalmologists and AI agreed on diagnosis in 48% cases, with 91.6% exhibiting concurrence in management plans. Conclusions: The study underscores AI chatbots' potential in uveitis diagnosis and management, indicating their value in reducing diagnostic errors. Further research is essential to enhance AI chatbot precision in diagnosis and recommendations.

AB - Introduction: Accurate diagnosis and timely management are vital for favorable uveitis outcomes. Artificial Intelligence (AI) holds promise in medical decision-making, particularly in ophthalmology. Yet, the diagnostic precision and management advice from AI-based uveitis chatbots lack assessment. Methods: We appraised diagnostic accuracy and management suggestions of an AI-based chatbot, ChatGPT, versus five uveitis-trained ophthalmologists, using 25 standard cases aligned with new Uveitis Nomenclature guidelines. Participants predicted likely diagnoses, two differentials, and next management steps. Comparative success rates were computed.Results: Ophthalmologists excelled (60–92%) in likely diagnosis, exceeding AI (60%). Considering fully and partially accurate diagnoses, ophthalmologists achieved 76–100% success; AI attained 72%. Despite an 8% AI improvement, its overall performance lagged. Ophthalmologists and AI agreed on diagnosis in 48% cases, with 91.6% exhibiting concurrence in management plans. Conclusions: The study underscores AI chatbots' potential in uveitis diagnosis and management, indicating their value in reducing diagnostic errors. Further research is essential to enhance AI chatbot precision in diagnosis and recommendations.

UR - https://www.scopus.com/pages/publications/85171542097

U2 - 10.1080/09273948.2023.2253471

DO - 10.1080/09273948.2023.2253471

M3 - Article

C2 - 37722842

AN - SCOPUS:85171542097

SN - 0927-3948

VL - 32

SP - 1526

EP - 1531

JO - Ocular Immunology and Inflammation

JF - Ocular Immunology and Inflammation

IS - 8

ER -

Evaluating the Diagnostic Accuracy and Management Recommendations of ChatGPT in Uveitis

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this