TY - JOUR
T1 - Systematic metareview of prediction studies demonstrates stable trends in bias and low PROBAST inter-rater agreement
AU - Langenhuijsen, Liselotte F.S.
AU - Janse, Roemer J.
AU - Venema, Esmee
AU - Kent, David M.
AU - van Diepen, Merel
AU - Dekker, Friedo W.
AU - Steyerberg, Ewout W.
AU - de Jong, Ype
N1 - Funding Information:
Funding: The work on this study by R.J.J. and M.v.D. was supported by a grant from the Dutch Kidney Foundation ( 20OK016 ).
Publisher Copyright:
© 2023 The Authors
PY - 2023/7
Y1 - 2023/7
N2 - Objectives: To (1) explore trends of risk of bias (ROB) in prediction research over time following key methodological publications, using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and (2) assess the inter-rater agreement of the PROBAST. Study Design and Setting: PubMed and Web of Science were searched for reviews with extractable PROBAST scores on domain and signaling question (SQ) level. ROB trends were visually correlated with yearly citations of key publications. Inter-rater agreement was assessed using Cohen's Kappa. Results: One hundred and thirty nine systematic reviews were included, of which 85 reviews (containing 2,477 single studies) on domain level and 54 reviews (containing 2,458 single studies) on SQ level. High ROB was prevalent, especially in the Analysis domain, and overall trends of ROB remained relatively stable over time. The inter-rater agreement was low, both on domain (Kappa 0.04–0.26) and SQ level (Kappa −0.14 to 0.49). Conclusion: Prediction model studies are at high ROB and time trends in ROB as assessed with the PROBAST remain relatively stable. These results might be explained by key publications having no influence on ROB or recency of key publications. Moreover, the trend may suffer from the low inter-rater agreement and ceiling effect of the PROBAST. The inter-rater agreement could potentially be improved by altering the PROBAST or providing training on how to apply the PROBAST.
AB - Objectives: To (1) explore trends of risk of bias (ROB) in prediction research over time following key methodological publications, using the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and (2) assess the inter-rater agreement of the PROBAST. Study Design and Setting: PubMed and Web of Science were searched for reviews with extractable PROBAST scores on domain and signaling question (SQ) level. ROB trends were visually correlated with yearly citations of key publications. Inter-rater agreement was assessed using Cohen's Kappa. Results: One hundred and thirty nine systematic reviews were included, of which 85 reviews (containing 2,477 single studies) on domain level and 54 reviews (containing 2,458 single studies) on SQ level. High ROB was prevalent, especially in the Analysis domain, and overall trends of ROB remained relatively stable over time. The inter-rater agreement was low, both on domain (Kappa 0.04–0.26) and SQ level (Kappa −0.14 to 0.49). Conclusion: Prediction model studies are at high ROB and time trends in ROB as assessed with the PROBAST remain relatively stable. These results might be explained by key publications having no influence on ROB or recency of key publications. Moreover, the trend may suffer from the low inter-rater agreement and ceiling effect of the PROBAST. The inter-rater agreement could potentially be improved by altering the PROBAST or providing training on how to apply the PROBAST.
UR - https://www.scopus.com/pages/publications/85162009198
U2 - 10.1016/j.jclinepi.2023.04.012
DO - 10.1016/j.jclinepi.2023.04.012
M3 - Article
C2 - 37142166
AN - SCOPUS:85162009198
SN - 0895-4356
VL - 159
SP - 159
EP - 173
JO - Journal of Clinical Epidemiology
JF - Journal of Clinical Epidemiology
ER -