Assessment of surgery residents' operative skills in the operating theater using a modified Objective Structured Assessment of Technical Skills (OSATS): A prospective multicenter study

Niels Hopmans, Ted Hoed, Luc van der Laan, E (Erwin) van der Harst, Marieke Bont, GHH Mannaerts, Imro Dawson, Reinier Timman, Bas Wijnhoven, J.N.M. IJzermans

Research output: Contribution to journalArticleAcademicpeer-review

77 Citations (Scopus)

Abstract

Background. With the implementation of competency-based curricula, Objective Structured Assessment of Technical Skills (OSATS) increasingly is being used for the assessment of operative skills. Although evidence for its usefulness has been demonstrated in experimental study designs, data supporting OSATS application in the operating room are limited. This study evaluates the validity and reliability of the OSATS instrument to assess the operative skills of surgery residents in the operating theater. Methods. Twenty-four residents were recruited from seven hospitals within a general surgical training region and classified equally into three groups according to postgraduate training year (PGY). Each resident had to perform five different types of operations. Surgical performance was measured using a modified OSATS consisting of three scales: Global Rating Scale, Overall Performance Scale, and Alphabetic Summary Scale. Validity and reliability metrics included construct validity (Kruskal-Wallis test) and internal consistency reliability (Cronbach's alpha coefficient). Spearman's correlation coefficients were calculated to determine correlations between the different scales. Results. Eighteen residents (PGY 1-2 [n = 7]; PGY 3-4 [n = 8]; PGY 5-6 [n = 3]) performed 249 operations. Comparisons of the performance scores revealed that evidence for construct validity depended on the difficulty level of the selected procedures. For individual operations, internal consistency reliability of the Global Rating Scale ranged from 0.93 to 0.95. Scores on the different scales correlated strongly (r = 0.62-0.76, P < .001). Conclusion. Assessment of operative skills in the operating theater using this modified OSATS instrument has the potential to establish learning curves, allowing adequate monitoring of residents' progress in achieving operative competence. The Alphabetic Summary Scale seems to be of additional value. Use of the Overall Performance Scale should be reconsidered.
Original languageUndefined/Unknown
Pages (from-to)1078-1088
Number of pages11
JournalSurgery
Volume156
Issue number5
DOIs
Publication statusPublished - 2014

Research programs

  • EMC MM-03-47-02-A
  • EMC NIHES-04-58-01

Cite this