Interrater Reliability and Diagnostic Performance of Subjective Evaluation of Sublingual Microcirculation Images by Physicians and Nurses: A Multicenter Observational Study

Introduction: This was a cross-sectional multicenter study to investigate the ability of physicians and nurses from three different countries to subjectively evaluate sublingual microcirculation images and thereby discriminate normal from abnormal sublingual microcirculation based on flow and density abnormalities. Methods: Forty-five physicians and 61 nurses (mean age, 36 +/- 10 years; 44 males) from three different centers in The Netherlands (n = 61), Uruguay (n = 12), and Japan (n = 33) were asked to subjectively evaluate a sample of 15 microcirculation videos randomly selected from an experimental model of endotoxic shock in pigs. All videos were first analyzed offline using the A.V.A. software by an independent, experienced investigator and were categorized as good, bad, or very bad microcirculation based on the microvascular flow index, perfused capillary density, and proportion of perfused capillaries. Then, the videos were randomly assigned to the examiners, who were instructed to subjectively categorize each image as good, bad, or very bad. An interrater analysis was performed, and sensitivity and specificity tests were calculated to evaluate the proportion of A.V.A. score abnormalities that the examiners correctly identified. Results: The kappa statistics indicated moderate agreement in the evaluation of microcirculation abnormalities using three categories, i.e., good, bad, or very bad (kappa = 0.48), and substantial agreement using two categories, i.e., normal (good) and abnormal (bad or very bad) (kappa = 0.66). There was no significant difference between the kappa three and kappa two statistics. We found that the examiner's subjective evaluations had good diagnostic performance and were highly sensitive (84%; 95% confidence interval, 81%-86%) and specific (87%; 95% confidence interval, 84%-90%) for sublingual microcirculatory abnormalities as assessed using the A.V.A. software. Conclusions: The subjective evaluations of sublingual microcirculation by physicians and nurses agreed well with a conventional offline analysis and were highly sensitive and specific for sublingual microcirculatory abnormalities.