Recalibration of prediction model was needed for monitoring health care quality in subgroups: a retrospective cohort study

Objectives: To evaluate the predictive ability of a mortality prediction model in subgroups of intensive care unit (ICU) patients and test the validity for monitoring the outcome. Study Design and Setting: A Japanese ICU database was used for the analyses. Adults admitted to an ICU between April 1, 2019, and March 31, 2020, were included. Nine clinically relevant subgroups were selected, and we evaluated the discrimination and calibration of the Japan Risk of Death model, a recalibrated Acute Physiology and Chronic Health Evaluation III-j model. Funnel plots and exponentially weighted moving average (EWMA) charts were used to check its validity for monitoring in-hospital mortality. If the predictive performance was poor, the model was recalibrated and model performance was reassessed. Results: The study population comprised 14,513 patients across nine subgroups. The in-hospital mortality rate ranged from 11.3% to 30.9%. The calibration was poor in most subgroups, and the funnel plots and EWMA charts frequently revealed “out-of-control” signals crossing the control limit of three standard deviations (SDs). The calibration improved after recalibration, and the number of “out-of-control” signals decreased. Conclusion: When monitoring the quality of care among subgroups of patients, testing the predictive ability and recalibration of the risk model are needed.