Effect of Predictive Model on ED Physician Assessments of Patient Disposition

NCT06434220 | Not Yet Recruiting

• 1 other identifier: IRB-P00048537
• Study Type: interventional
• Target: 10
• Locations: 0 countries
• Sites: N/A

Brief Summary

The goal of this study is to measure the impact of fairness-aware algorithms on physician predictions of ED patient admission. Using an experimentally validated machine learning model tuned for equitable outcomes, the investigators quantify the impact of model recommendations on ED physician assessments of admission risk in a silent, prospective study. The investigators survey ED physicians who are not currently caring for patients using live site data. To quantify the impact of the model on ED physician assessments of admission risk, the investigators collect physician assessments before and after consulting the (original or updated) model prediction. The investigators measure ED physician adherence to model suggestions, along with the predictive accuracy and equity of downstream patient outcomes. The outcome of this study is an empirical measure of the extent to which fair ML models may influence admission decisions to mitigate health care disparities.

Conditions

Patient Outcome Assessment

Outcome Measures

Primary Outcomes (2)

• Physician-assessed ED disposition (likelihood of admission)

  The primary outcome is physician-assessed ED disposition (categorized as admission or discharge), before and after viewing a model prediction, compared to final disposition of patient

  Within 24 hours of survey

• Patient final disposition (admitted/discharged)

  The final disposition of the patient, whether admitted to an inpatient service or discharged

  Within 24 hours of survey

Secondary Outcomes (1)

• Model-assessed ED disposition

  Within 24 hours of survey

Study Arms (3)

Physician assessment before intervention

NO INTERVENTION

No intervention. Physician is surveyed to provide their assessment of patient disposition.

Physician assessment after baseline model intervention

ACTIVE COMPARATOR

Physician is shown a baseline model recommendation for patient disposition including description of factors driving the model prediction.

Diagnostic Test: Baseline model

Physician assessment after fairness-aware model intervention

ACTIVE COMPARATOR

Physician is shown a model recommendation form a model tuned for subgroup performance for patient disposition including description of factors driving the model prediction.

Diagnostic Test: Fairness-aware model

Interventions

Fairness-aware model DIAGNOSTIC_TEST

Model prediction of patient disposition including feature importance scores driving prediction. This model has been tuned to minimize subgroup calibration errors.

Physician assessment after fairness-aware model intervention

Baseline model DIAGNOSTIC_TEST

Model prediction of patient disposition including feature importance scores driving prediction.

Physician assessment after baseline model intervention

Eligibility Criteria

• Age: 18 Years - 65 Years
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may qualify if:

• Board certified emergency department attending physicians currently employed by Boston Children's Hospital

You may not qualify if:

• Physicians are excluded from completely surveys for patients who they are currently caring for

Sponsors & Collaborators

• Boston Children's Hospital: lead

Related Publications (5)

• Barak-Corren Y, Israelit SH, Reis BY. Progressive prediction of hospitalisation in the emergency department: uncovering hidden patterns to improve patient flow. Emerg Med J. 2017 May;34(5):308-314. doi: 10.1136/emermed-2014-203819. Epub 2017 Feb 10.

  PMID: 28188202 BACKGROUND

• Barak-Corren Y, Agarwal I, Michelson KA, Lyons TW, Neuman MI, Lipsett SC, Kimia AA, Eisenberg MA, Capraro AJ, Levy JA, Hudgins JD, Reis BY, Fine AM. Prediction of patient disposition: comparison of computer and human approaches and a proposed synthesis. J Am Med Inform Assoc. 2021 Jul 30;28(8):1736-1745. doi: 10.1093/jamia/ocab076.

  PMID: 34010406 BACKGROUND

• Barak-Corren Y, Chaudhari P, Perniciaro J, Waltzman M, Fine AM, Reis BY. Prediction across healthcare settings: a case study in predicting emergency department disposition. NPJ Digit Med. 2021 Dec 15;4(1):169. doi: 10.1038/s41746-021-00537-x.

  PMID: 34912043 BACKGROUND

• Barak-Corren Y, Fine AM, Reis BY. Early Prediction Model of Patient Hospitalization From the Pediatric Emergency Department. Pediatrics. 2017 May;139(5):e20162785. doi: 10.1542/peds.2016-2785.

  PMID: 28557729 BACKGROUND

• La Cava WG, Lett E, Wan G. Fair admission risk prediction with proportional multicalibration. Proc Mach Learn Res. 2023;209:350-378.

  PMID: 37576024 BACKGROUND

Central Study Contacts

William La Cava, PhD

CONTACT

4133200544; william.lacava@childrens.harvard.edu

Andrew Fine, MD

CONTACT

617-355-9696; andrew.fine@childrens.harvard.edu

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: TRIPLE
• Who Masked: PARTICIPANT, CARE PROVIDER, INVESTIGATOR
• Masking Details: We conduct a randomized, double-blind, controlled before-after (CBA) study of board-certified ED attending physicians not currently caring for patients in the BCH ED over a period of six to twelve weeks. In this experiment, the "treatment" consists of an ML recommendation provided to the ED physicians, who predict admission decisions for individual patients before and after receiving it. The "control" surveys receive the original POPP model recommendation, and "treatment" surveys receive a "fairness-aware" model, determined in prior work to mitigate biases in performance with respect to patient demographics
• Purpose: OTHER
• Intervention Model: SEQUENTIAL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Assistant Professor

Study Record Dates

• First Submitted: May 20, 2024
• First Posted: May 30, 2024
• Study Start (Estimated): January 1, 2027
• Primary Completion (Estimated): May 1, 2027
• Study Completion (Estimated): September 1, 2027
• Last Updated: April 13, 2026

Record last verified: 2026-04

Data Sharing

• IPD Sharing: Will not share