The Impact of Large Language Models on Diagnostic Reasoning Among LLM-Trained Medical Doctors

NCT06774612 | Completed

• 1 other identifier: IRB-0342
• Study Type: interventional
• Target: 60
• Locations: 1 country
• Sites: 1

Brief Summary

This study aims to evaluate whether large language model-trained medical doctors demonstrate enhanced diagnostic reasoning performance when utilizing ChatGPT-4o alongside conventional resources compared to using conventional resources alone.

Conditions

Diagnosis

Keywords

clinical reasoning; large language models; computer-assisted diagnosis

Outcome Measures

Primary Outcomes (1)

• Diagnostic reasoning

  The primary outcome will be the percent correct for each case (range: 0 to 100). For each case, participants will be asked for three top diagnoses, findings from the case that support that diagnosis, and findings from the case that oppose that diagnosis. For each plausible diagnosis, participants will receive 1 point. Findings supporting the diagnosis and findings opposing the diagnosis will also be graded based on correctness, with 1 point for partially correct and 2 points for completely correct responses. Participants will then be asked to name their top diagnosis, earning one point for a reasonable response and two points for the most correct response. Finally participants will be asked to name up to 3 next steps to further evaluate the patient with one point awarded for a partially correct response and two points for a completely correct response. The primary outcome will be compared on the case-level by the randomized groups.

  Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.

Secondary Outcomes (1)

• Time Spent on Diagnosis

  Assessed at a single time point for each case, during the scheduled diagnostic reasoning evaluation session, which takes place between 0-4 days after participant enrollment.

Study Arms (2)

ChatGPT-4o

ACTIVE COMPARATOR

Group will be given access to ChatGPT-4o.

Other: ChatGPT-4o

Conventional resources

NO INTERVENTION

Group will not be given access to ChatGPT-4o but will be encouraged to use any resources they wish besides large language models (PubMed, Google without AI Overviews, etc).

Interventions

ChatGPT-4o OTHER

OpenAI's ChatGPT-4o large language model with chat interface.

ChatGPT-4o

Eligibility Criteria

• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)

You may qualify if:

• Full or Provisionally Registered Medical Practitioners with the Pakistan Medical and Dental Council (PMDC).
• Completed Bachelor of Medicine, Bachelor of Surgery (MBBS) Exam. The equivalent degree of MBBS in US and Canada is called Doctor of Medicine (MD).
• Participants must have completed a structured training program on the use of ChatGPT (or a comparable large language model), totaling at least 10 hours of instruction. The program must include hands-on practice related to LLM's aspects, specifically prompt engineering and content evaluation.

You may not qualify if:

• Any other Registered Medical Practitioners (Full or Provisional) with PMDC (e.g., Professionals with Bachelor of Dental Surgery or BDS).

Sponsors & Collaborators

• Lahore University of Management Sciences: lead
• King Edward Medical University: collaborator

Study Sites (1)

Lahore University of Management Sciences

Lahore, Punjab Province, 54792, Pakistan

Location

MeSH Terms

Conditions

Disease

Condition Hierarchy (Ancestors)

Pathologic Processes; Pathological Conditions, Signs and Symptoms

Study Officials

• Ihsan Ayyub Qazi, PhD

  Lahore University of Management Sciences

  PRINCIPAL INVESTIGATOR

• Muhammad Asadullah Khawaja, MBBS

  King Edward Medical University

  PRINCIPAL INVESTIGATOR

• Ayesha Ali, PhD

  Lahore University of Management Sciences

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: NONE
• Masking Details: Single (Outcomes Assessor)
• Purpose: DIAGNOSTIC
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Associate Professor, PhD

Model Details: The trial will be designed as a randomized, two-arm, single-blind parallel group study.

Study Record Dates

• First Submitted: January 4, 2025
• First Posted: January 14, 2025
• Study Start: January 10, 2025
• Primary Completion: May 17, 2025
• Study Completion: May 17, 2025
• Last Updated: July 17, 2025

Record last verified: 2025-07

Data Sharing

• IPD Sharing: Will not share

Locations

PA (1)