Simple dichotomous updating methods improved the validity of polytomous prediction models

Objectives: Prediction models may perform poorly in a new setting. We aimed to determine which model updating methods should be applied for models predicting polytomous outcomes, which often suffer from one or more categories with low prevalence. Study Design and Setting: We used case studies on testicular and ovarian tumors. The original regression models were based on 502 and 2,037 patients and validated on 273 and 1,107 patients, respectively. The polytomous models combined dichotomous models for category A vs. B + C and B vs. C (sequential dichotomous modeling). Simple recalibration, revision, and redevelopment methods were considered. To assess discrimination (using dichotomous and polytomous c-statistics) and calibration (by compar Results: None of the updating methods could improve discrimination of the original models, but recalibration, revision, and redevelopment strongly improved calibration. Redevelopment was unstable with respect to overfitting and performance. Conclusion: Simple dichotomous updating methods behaved well when applied to polytomous models. Our results suggest that recalibration is preferred, but larger validation sets may make revision or redevelopment a sensible alternative. (C) 2013 Elsevier Inc. All rights reserved.