TY - JOUR
AU - K. Nguyen
AU - D. L. Wilson
AU - J. Diiulio
AU - B. Hall
AU - L. Militello
AU - W. F. Gellad
AU - C. A. Harle
AU - M. Lewis
AU - S. Schmidt
AU - E. I. Rosenberg
AU - D. Nelson
AU - X. He
AU - Y. Wu
AU - J. Bian
AU - S. A. S. Staras
AU - A. J. Gordon
AU - J. Cochran
AU - C. Kuza
AU - S. Yang
AU - W. Lo-Ciganic
A1 - 
AB - BACKGROUND: Integrating advanced machine-learning (ML) algorithms into clinical practice is challenging and requires interdisciplinary collaboration to develop transparent, interpretable, and ethically sound clinical decision support (CDS) tools. We aimed to design a ML-driven CDS tool to predict opioid overdose risk and gather feedback for its integration into the University of Florida Health (UFHealth) electronic health record (EHR) system. METHODS: We used user-centered design methods to integrate the ML algorithm into the EHR system. The backend and UI design sub-teams collaborated closely, both informed by user feedback sessions. We conducted seven user feedback sessions with five UF Health primary care physicians (PCPs) to explore aspects of CDS tools, including workflow, risk display, and risk mitigation strategies. After customizing the tool based on PCPs' feedback, we held two rounds of one-on-one usability testing sessions with 8 additional PCPs to gather feedback on prototype alerts. These sessions informed iterative UI design and backend processes, including alert frequency and reappearance circumstances. RESULTS: The backend process development identified needs and requirements from our team, information technology, UFHealth, and PCPs. Thirteen PCPs (male = 62%, White = 85%) participated across 7 user feedback sessions and 8 usability testing sessions. During the user feedback sessions, PCPs (n = 5) identified flaws such as the term "high risk" of overdose potentially leading to unintended consequences (e.g., immediate addiction services referrals), offered suggestions, and expressed trust in the tool. In the first usability testing session, PCPs (n = 4) emphasized the need for natural risk presentation (e.g., 1 in 200) and suggested displaying the alert multiple times yearly for at-risk patients. Another 4 PCPs in the second usability testing session valued the UFHealth-specific alert for managing new or unfamiliar patients, expressed concerns about PCPs' workload when prescribing to high-risk patients, and recommended incorporating the details page into training sessions to enhance usability. CONCLUSIONS: The final backend process for our CDS alert aligns with PCP needs and UFHealth standards. Integrating feedback from PCPs in the early development phase of our ML-driven CDS tool helped identify barriers and facilitators in the CDS integration process. This collaborative approach yielded a refined prototype aimed at minimizing unintended consequences and enhancing usability.
AD - Department of Pharmacotherapy and Translational Research, College of Pharmacy, University of Florida, Gainesville, FL, USA.; Department of Pharmaceutical Outcomes and Policy, College of Pharmacy, University of Florida, Gainesville, FL, USA.; Applied Decision Science, Cincinnati, OH, USA.; Division of General Internal Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA.; Center for Pharmaceutical Policy and Prescribing, University of Pittsburgh, Pittsburgh, PA, USA.; Center for Health Equity Research Promotion, Veterans Affairs Pittsburgh Healthcare System, Pittsburgh, PA, USA.; Department of Health Policy and Management, Richard M. Fairbanks School of Public Health, Indiana University, Indianapolis, IN, USA.; Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, IN, USA.; Department of Community Health and Family Medicine, College of Medicine, University of Florida, Gainesville, FL, USA.; Division of General Internal Medicine, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA.; Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA.; Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, UT, USA.; Informatics, Decision-Enhancement, and Analytic Sciences Center, Veterans Administration Salt Lake City Health Care System, Salt Lake City, UT, USA.; Division of General Internal Medicine, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA. jenny.lociganic@pitt.edu.; Center for Pharmaceutical Policy and Prescribing, University of Pittsburgh, Pittsburgh, PA, USA. jenny.lociganic@pitt.edu.; Geriatric Research Education and Clinical Center, North Florida/South Georgia Veterans Health System, Gainesville, FL, USA. jenny.lociganic@pitt.edu.
AN - 39420438
BT - Bioelectron Med
C5 - Opioids & Substance Use; HIT & Telehealth
CP - 1
DA - Oct 18
DO - 10.1186/s42234-024-00156-3
DP - NLM
ET - 20241018
IS - 1
JF - Bioelectron Med
LA - eng
N2 - BACKGROUND: Integrating advanced machine-learning (ML) algorithms into clinical practice is challenging and requires interdisciplinary collaboration to develop transparent, interpretable, and ethically sound clinical decision support (CDS) tools. We aimed to design a ML-driven CDS tool to predict opioid overdose risk and gather feedback for its integration into the University of Florida Health (UFHealth) electronic health record (EHR) system. METHODS: We used user-centered design methods to integrate the ML algorithm into the EHR system. The backend and UI design sub-teams collaborated closely, both informed by user feedback sessions. We conducted seven user feedback sessions with five UF Health primary care physicians (PCPs) to explore aspects of CDS tools, including workflow, risk display, and risk mitigation strategies. After customizing the tool based on PCPs' feedback, we held two rounds of one-on-one usability testing sessions with 8 additional PCPs to gather feedback on prototype alerts. These sessions informed iterative UI design and backend processes, including alert frequency and reappearance circumstances. RESULTS: The backend process development identified needs and requirements from our team, information technology, UFHealth, and PCPs. Thirteen PCPs (male = 62%, White = 85%) participated across 7 user feedback sessions and 8 usability testing sessions. During the user feedback sessions, PCPs (n = 5) identified flaws such as the term "high risk" of overdose potentially leading to unintended consequences (e.g., immediate addiction services referrals), offered suggestions, and expressed trust in the tool. In the first usability testing session, PCPs (n = 4) emphasized the need for natural risk presentation (e.g., 1 in 200) and suggested displaying the alert multiple times yearly for at-risk patients. Another 4 PCPs in the second usability testing session valued the UFHealth-specific alert for managing new or unfamiliar patients, expressed concerns about PCPs' workload when prescribing to high-risk patients, and recommended incorporating the details page into training sessions to enhance usability. CONCLUSIONS: The final backend process for our CDS alert aligns with PCP needs and UFHealth standards. Integrating feedback from PCPs in the early development phase of our ML-driven CDS tool helped identify barriers and facilitators in the CDS integration process. This collaborative approach yielded a refined prototype aimed at minimizing unintended consequences and enhancing usability.
PY - 2024
SN - 2332-8886
SP - 24
ST - Design and development of a machine-learning-driven opioid overdose risk prediction tool integrated in electronic health records in primary care settings
T1 - Design and development of a machine-learning-driven opioid overdose risk prediction tool integrated in electronic health records in primary care settings
T2 - Bioelectron Med
TI - Design and development of a machine-learning-driven opioid overdose risk prediction tool integrated in electronic health records in primary care settings
U1 - Opioids & Substance Use; HIT & Telehealth
U3 - 10.1186/s42234-024-00156-3
VL - 10
VO - 2332-8886
Y1 - 2024
ER -