Generation of preoperative anaesthetic plans by ChatGPT-4.0: a mixed-method study

BACKGROUND: Recent advances in artificial intelligence (AI) have enabled development of natural language algorithms capable of generating coherent texts. We evaluated the quality, validity, and safety of this generative AI in preoperative anaesthetic planning.

METHODS: In this exploratory, single-centre, convergent mixed-method study, 10 clinical vignettes were randomly selected, and ChatGPT (OpenAI, 4.0) was prompted to create anaesthetic plans, including cardiopulmonary risk assessment, intraoperative anaesthesia technique, and postoperative management. A quantitative assessment compared these plans with those made by eight senior anaesthesia consultants. A qualitative assessment was performed by an adjudication committee through focus group discussion and thematic analysis. Agreement on cardiopulmonary risk assessment was calculated using weighted Kappa, with descriptive data representation for other outcomes.

RESULTS: ChatGPT anaesthetic plans showed variable agreement with consultants' plans. ChatGPT, the survey panel, and adjudication committee frequently disagreed on cardiopulmonary risk estimation. The ChatGPT answers were repetitive and lacked variety, evidenced by the strong preference for general anaesthesia and absence of locoregional techniques. It also showed inconsistent choices regarding airway management, postoperative analgesia, and medication use. While some differences were not deemed clinically significant, subpar postoperative pain management advice and failure to recommend tracheal intubation for patients at high risk for pulmonary aspiration were considered inappropriate recommendations.

CONCLUSIONS: Preoperative anaesthetic plans generated by ChatGPT did not consistently meet minimum clinical standards and were unlikely the result of clinical reasoning. Therefore, ChatGPT is currently not recommended for preoperative planning. Future large language models trained on anaesthesia-specific datasets might improve performance but should undergo vigorous evaluation before use in clinical practice.