Assessing the utility and challenges for implementation of a risk prediction system: a usability study with hospital pharmacists

Background: The clinical implementation of prediction models can face important barriers, particularly regarding user-friendliness and interpretability for healthcare professionals. We recently developed and externally validated a risk prediction model for denosumab-induced hypocalcemia. The present study aimed to evaluate the model’s utility and identify challenges for its clinical implementation through pilot testing conducted by hospital pharmacists. Methods: A paper-format prediction model was distributed to pharmacists at Kameda General Hospital, Miyagi Cancer Center, and Gunma Prefectural Cancer Center. Participants trialed the model outside their routine workflow by applying it to data from patients scheduled to receive their first dose of denosumab. A subsequent questionnaire survey, available in paper and electronic formats, was conducted to gather feedback on the model’s utility and limitations. Results: A total of 49 responses were obtained, predominantly from pharmacists in their 20 s and 30 s with diverse professional responsibilities. The model was positively evaluated in terms of its clear target population, predicted outcome (47/49, 95.9%), and simple, straightforward nature (47/49, 95.9%). However, some participants provided neutral feedback on its ease of use (10/49, 20.4%). While its potential as an auxiliary tool for risk prediction was acknowledged, there were also some neutral views on its practical utility. One concern was the inconvenience of implementing a paper-format tool in clinical environments that primarily operate on electronic health records. Conclusions: The paper-format prediction model was positively evaluated by frontline pharmacists, especially for its clear and straightforward nature. As anticipated, however, limitations such as manual data input and paper-based format were identified by some participants. Integration into electronic health records and broader clinical validation will be necessary to advance the clinical application of this prediction model and ensure real-world applicability.