Can Feedback From a Large Language Model Improve Health Care Quality?

NCT06823765 | Completed

• 1 other identifier: 2000035990
• Study Type: interventional
• Target: 491
• Locations: 1 country
• Sites: 2

Brief Summary

The goal of this study is to learn if computer-assisted advice can help improve patient care in Nigerian health clinics. The main question it aims to answer is: does giving healthcare workers instant computer feedback help them make better decisions about patient care? Researchers will compare patient care notes written by healthcare workers before and after they receive computer feedback to see if the feedback improves care quality. A doctor who doesn't know if feedback was given will review these notes. Participants will:

• Be seen by a community healthcare worker who uses the computer feedback system
• Be treated by a fully trained medical doctor
• Get tested for malaria, anemia, or urinary tract infections if they have certain symptoms

Conditions

All Conditions

Keywords

Large Language Models; Primary Care; Nigeria

Outcome Measures

Primary Outcomes (4)

• Indicator for an Error in the Treatment plan (with the Potential for Harm)

  During SOAP note evaluation, the MO is asked to indicate whether the treatment plan for the patient contains any errors, conditional on the MO's own diagnosis. This is coded as 1 if the MO indicates there is an error and 0 otherwise. The introductory text (here for SOAP Note A) is: Please evaluate whether the treatment in SOAP Note A is appropriate for this patient's condition. Please base this on your own diagnosis, not the CHEW's diagnosis in SOAP Note A. This is followed by the question: Is the treatment plan for the patient in SOAP Note A completely appropriate given your own diagnosis (accounting for conditional treatments based on medical tests)? Answer "No" if the patient should receive different medical care given your diagnosis. This can include both minor differences (for example, the patient should be advised to rest) and major errors (for example, the patient should receive a completely different set of medications). (Answer options: yes/no/unsure)

  Through study completion, an average of six months

• Indicator for an Error in the Treatment Plan that Causes a Loss of at least X Quality-Adjusted Life Days

  This variable is coded as 1 if the MO indicates there is such an error and 0 otherwise. X is defined to be the highest benchmark on the appropriate DALY scale so that at least 5% of patients have an error that large in the unassisted SOAP note. In other words, severe errors are any errors that generate a harm rating at or above the 95th percentile of harm on the unassisted scale (pooling child and adult scales).

  Through study completion, an average of six months

• Indicator for the Better Treatment Plan (as Determined by the MOs)

  Based on the DALY rating of SOAP Note A vs. B (counting instances with no errors as 0 DALY loss), the indicator is coded as 1 if the SOAP note has the better treatment plan (lower DALY loss) and 0 if MOs judge both notes to be the same in response to the following question: Are there any meaningful differences in the treatment plans of SOAP Note A and B?

  Through study completion, an average of six months

• Indicator for whether Treatment is Consistent with a Predetermined "Standard of Care"

  At-risk patients receive malaria, anemia and UTI screening in accordance with certain demographic criteria. A dataset is then constructed with one observation for each (patient, screening test, note), up to six per patient. The indicator of treatment misallocation records whether a patient was incorrectly treated for a condition based on the test result or lack of symptoms. The variable is coded as 1 if the patient tested positive and either received inappropriate or no treatment. It is also coded as 1 if the patient tested negative or was not tested based on the symptom screen but received treatment for the condition. The variable is only coded as 0 if the patient tested negative and was correctly not treated for the corresponding condition, or if they tested positive and received the correct treatment.

  Through study completion, an average of six months

Secondary Outcomes (12)

• Indicators Denoting Diagnosis and Treatment Alignment Between CHEWs and MOs

  Through study completion, an average of six months

• Alternative Indicators for Treatment Misallocation

  Through study completion, an average of six months

• Relationship of QALY Loss to Severity of Patient Condition

  Through study completion, an average of six months

• Indicators for the Appropriateness of Medical Testing Decisions

  Through study completion, an average of six months

• Average and Distribution of DALY Lost

  Through study completion, an average of six months

Study Arms (1)

Clinical Assessment with and without LLMs

EXPERIMENTAL

The investigators employ a within-patient design. Patients receive two sequential assessments from a Community Health Extension Worker: first without and then with Large Language Model assistance.

Other: Large Language Model Clinical Decision Support

Interventions

Large Language Model Clinical Decision Support OTHER

A Large Language Model (LLM) integrated into the clinic's Electronic Medical Record system provides real-time feedback on patient assessments. Community Health Extension Workers first create a standard SOAP note, submit it to the LLM, and receive detailed feedback and key recommendations. They can then update their assessment based on this feedback. All final treatment decisions are made by Medical Officers who independently evaluate patients.

Clinical Assessment with and without LLMs

Eligibility Criteria

• Sex: all
• Healthy Volunteers: No
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)

You may qualify if:

• Patient is at the clinic for outpatient consultation
• Parent/guardian consent is required for individuals under 18

You may not qualify if:

• Patient does not require emergency care
• Patient is not at the clinic for a checkup (e.g. weight, blood pressure, follow up after recovery)
• Patient is not a trauma patient (visit is not for an accident, wound or injury)
• Patient is not at the clinic for a scheduled procedure or a birth

Sponsors & Collaborators

• Yale University: lead
• EHA Clinics Nigeria: collaborator
• World Bank: collaborator
• University of Pennsylvania: collaborator
• George Washington University: collaborator

Study Sites (2)

EHA Clinics REACH Community Clinic, Gyadi Gyadi

Kano, Kano State, Nigeria

Location

EHA Clinics, 33 Lamido Crescent

Kano, Kano State, Nigeria

Location

Study Officials

• Jason Abaluck

  Yale University

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: NA
• Masking: NONE
• Masking Details: The Medical Officers and the panel of Medical Doctors are both blinded to which note has been generated with LLM assistance.
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: SINGLE GROUP
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Professor of Economics, Yale School of Management

Model Details: For the main analysis, the study employs a within-patient comparison of two patient notes created by the CHEW; one during the initial patient consultation, and one after the LLM feedback was received. The patient is also seen by a fully trained medical officer who is in charge of patient care. The MO conducts a blinded review of the CHEW's patient notes to measures changes in the CHEW's care as a result of the LLM feedback. The data comes from the information captured in the electronic medical record (EMR) of the patient and from survey data collected from CHEWs, reviewing MOs, and a panel of reviewing Medical Doctors.

Study Record Dates

• First Submitted: January 23, 2025
• First Posted: February 12, 2025
• Study Start: January 30, 2025
• Primary Completion: October 17, 2025
• Study Completion: October 17, 2025
• Last Updated: February 3, 2026

Record last verified: 2025-04

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ICF, CSR, ANALYTIC CODE
• Time Frame: Data will be available to other researchers beginning 3 months after publication and will remain available with no end date.
• Access Criteria: Academic researchers with a formal appointment at a research institution must submit a research proposal detailing intended analyses and sign a data use agreement.

Locations

NG (2)