External validation of an artificial intelligence–based model for retinopathy of prematurity screening using Phoenix ICON retinal images

Purpose: 

To assess the performance of a RetCam-trained artificial intelligence (AI) algorithm for the autonomous detection of severe retinopathy of prematurity (ROP) using retinal images acquired with the smaller field-of-view Phoenix ICON retinal camera. 

Methods: 

Retrospective external validation was performed using Phoenix ICON retinal images captured during ROP screening examinations in a Dutch cohort of infants born in 2021. Images of insufficient quality were excluded via automated quality assessment. Model performances for more-than-mild ROP (MTM-ROP)—type 1 or 2 ROP, or any ROP with pre-plus disease—and for type 1 ROP alone, were expressed as area under the precision-recall curve (AUPRC), sensitivity and specificity. 

Results: 

A total of 4,411 images from 66 infants were captured during 419 individual eye examinations, averaging 67 ± 65 images per infant and 10 ± 6 images per eye examination. Sixty examinations (14.3%) had all images excluded in automated quality assessment. When using the best performance between both eyes to assess infant-level performance, AUPRC was 0.911 (95% CI, 0.638-1.000), sensitivity was 82.0% (95% CI, 73.0-89.0) and specificity was 77.0% (95% CI, 68.1-84.4) for MTM-ROP. For type 1 ROP alone, AUPRC was 0.983 (95% CI, 0.964-1.000), sensitivity was 100.0% (95% CI, 94.7-100.0), and specificity was 72.4% (95% CI, 64.4-79.5). 

Conclusions: 

The algorithm's performance with Phoenix ICON is similar to its performance with RetCam. All infants with treatment-requiring type 1 ROP were detected by the algorithm. The presence of eye examinations without images of sufficient quality underlines the need for imaging protocols, especially when using this algorithm, with a smaller field-of-view camera.