TY - JOUR
AU - S. A. Hooker
AU - R. C. Rossom
AU - J. Haapala
AU - A. L. Crain
AU - K. Miley
AU - A. W. Olson
AU - S. Kane
AU - K. Christensen
AU - P. J. O'Connor
AU - G. Bart
AU - E. A. Wright
A1 - 
AB - BACKGROUND: Most people with opioid use disorder (OUD) do not receive evidence-based treatment. To increase treatment rates, primary care clinics may choose to implement risk prediction tools available in the electronic health record (EHR) to identify patients with a high risk of OUD or overdose. OBJECTIVE: To externally validate Epic's cognitive computing model to predict the Risk of Opioid Abuse or Overdose (referred to as the Opioid Risk Score; ORS) in three large integrated health systems. DESIGN: Prospective cohort study secondary to an ongoing clinical trial. PARTICIPANTS: Patients (N = 704,764) aged 18-75 who had a primary care encounter during the study period (April 2021-December 2022) and did not have an OUD diagnosis at index. MAIN MEASURES: Data were extracted from the EHR. The index date was defined as the first date within the study period where the patient met eligibility criteria and had an ORS calculated by the EHR. The binary outcome variable was whether the patient was diagnosed with OUD or experienced an opioid overdose within 12 months of the index date. KEY RESULTS: Most patients were classified as low risk on ORS (99.6%). Few patients experienced an OUD diagnosis or overdose in the 12-month follow-up period (0.3%). The model correctly classified 185 of 2362 patients who experienced an event (sensitivity 0.0783, 95% CI 0.0675, 0.0892) and 699,926 of 702,406 patients who did not experience an event (specificity 0.9965, 95% CI 0.9963, 0.9966). Few patients with high ORS experienced the event (PPV 0.0694, 95% CI 0.0598, 0.0791). The model had excellent discrimination (c-statistic = 0.815) but was poorly calibrated, underestimating risk for patients who experienced the outcomes. CONCLUSIONS: Epic's ORS demonstrated excellent discrimination but very low sensitivity across three large integrated health systems. Health systems should exercise caution before implementing vendor risk prediction models without validating their use in their patient populations. CLINICAL TRIAL NUMBER: Not applicable.
AD - Research and Evaluation Division, HealthPartners Institute, Minneapolis, MN, USA. stephanie.a.hooker@healthpartners.com.; Research and Evaluation Division, HealthPartners Institute, Minneapolis, MN, USA.; Essentia Institute of Rural Health, Duluth, MN, USA.; Geisinger, Center for Pharmacy Innovation and Outcomes, Danville, PA, USA.; Hennepin Healthcare, Minneapolis, MN, USA.; Geisinger, Department of Bioethics and Decision Sciences, Danville, PA, USA.
AN - 41703383
BT - J Gen Intern Med
C5 - Opioids & Substance Use; HIT & Telehealth
DA - Feb 17
DO - 10.1007/s11606-026-10257-1
DP - NLM
ET - 20260217
JF - J Gen Intern Med
LA - eng
N2 - BACKGROUND: Most people with opioid use disorder (OUD) do not receive evidence-based treatment. To increase treatment rates, primary care clinics may choose to implement risk prediction tools available in the electronic health record (EHR) to identify patients with a high risk of OUD or overdose. OBJECTIVE: To externally validate Epic's cognitive computing model to predict the Risk of Opioid Abuse or Overdose (referred to as the Opioid Risk Score; ORS) in three large integrated health systems. DESIGN: Prospective cohort study secondary to an ongoing clinical trial. PARTICIPANTS: Patients (N = 704,764) aged 18-75 who had a primary care encounter during the study period (April 2021-December 2022) and did not have an OUD diagnosis at index. MAIN MEASURES: Data were extracted from the EHR. The index date was defined as the first date within the study period where the patient met eligibility criteria and had an ORS calculated by the EHR. The binary outcome variable was whether the patient was diagnosed with OUD or experienced an opioid overdose within 12 months of the index date. KEY RESULTS: Most patients were classified as low risk on ORS (99.6%). Few patients experienced an OUD diagnosis or overdose in the 12-month follow-up period (0.3%). The model correctly classified 185 of 2362 patients who experienced an event (sensitivity 0.0783, 95% CI 0.0675, 0.0892) and 699,926 of 702,406 patients who did not experience an event (specificity 0.9965, 95% CI 0.9963, 0.9966). Few patients with high ORS experienced the event (PPV 0.0694, 95% CI 0.0598, 0.0791). The model had excellent discrimination (c-statistic = 0.815) but was poorly calibrated, underestimating risk for patients who experienced the outcomes. CONCLUSIONS: Epic's ORS demonstrated excellent discrimination but very low sensitivity across three large integrated health systems. Health systems should exercise caution before implementing vendor risk prediction models without validating their use in their patient populations. CLINICAL TRIAL NUMBER: Not applicable.
PY - 2026
SN - 0884-8734
ST - External Validation of Epic's Risk of Opioid Abuse and Overdose Model Among Primary Care Patients in Three Health Systems
T1 - External Validation of Epic's Risk of Opioid Abuse and Overdose Model Among Primary Care Patients in Three Health Systems
T2 - J Gen Intern Med
TI - External Validation of Epic's Risk of Opioid Abuse and Overdose Model Among Primary Care Patients in Three Health Systems
U1 - Opioids & Substance Use; HIT & Telehealth
U3 - 10.1007/s11606-026-10257-1
VO - 0884-8734
Y1 - 2026
ER -