Physician Reasoning on Management Cases With Large Language Models

NCT06208423 | Completed

• 1 other identifier: 71319b
• Study Type: interventional
• Target: 92
• Locations: 1 country
• Sites: 1

Brief Summary

This study will evaluate the effect of providing access to GPT-4, a large language model, compared to traditional management decision support tools on performance on case-based management reasoning tasks.

Conditions

Clinical Decision-making

Keywords

clinical decision support

Outcome Measures

Primary Outcomes (1)

• Management Reasoning

  Percent correct (range: 0 to 100) for each case.

  Within one-hour study

Secondary Outcomes (1)

• Time Spent on Management

  Within one-hour study

Study Arms (2)

GPT-4

ACTIVE COMPARATOR

Group will be given access to GPT-4

Other: GPT-4

Usual Resources

NO INTERVENTION

Group will not be given access to GPT-4 but will be encouraged to use any resources they wish besides large language models (UpToDate, Dynamed, google, etc).

Interventions

GPT-4 OTHER

OpenAI's GPT-4 large language model with chat interface.

GPT-4

Eligibility Criteria

• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)

You may qualify if:

• Participants must be licensed physicians and have completed at least post-graduate year 2 (PGY2) of medical training.
• Training in Internal medicine, family medicine, or emergency medicine.

You may not qualify if:

• Not currently practicing clinically.

Sponsors & Collaborators

• Stanford University: lead
• Beth Israel Deaconess Medical Center: collaborator
• University of Minnesota: collaborator

Study Sites (1)

Stanford University

Palo Alto, California, 94304, United States

Location

Related Publications (2)

• Goh E, Gallo RJ, Strong E, Weng Y, Kerman H, Freed JA, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. GPT-4 assistance for improvement of physician performance on patient care tasks: a randomized controlled trial. Nat Med. 2025 Apr;31(4):1233-1238. doi: 10.1038/s41591-024-03456-y. Epub 2025 Feb 5.

  PMID: 39910272 DERIVED

• Goh E, Gallo R, Strong E, Weng Y, Kerman H, Freed J, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. medRxiv [Preprint]. 2024 Aug 7:2024.08.05.24311485. doi: 10.1101/2024.08.05.24311485.

  PMID: 39148822 DERIVED

Study Officials

• Jonathan H Chen, MD, PhD

  Stanford University

  PRINCIPAL INVESTIGATOR

• Adam Rodman, MD

  Beth Israel Deaconess Medical Center

  PRINCIPAL INVESTIGATOR

• Andrew Olson, MD

  University of Minnesota

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: SINGLE
• Who Masked: OUTCOMES ASSESSOR
• Masking Details: The grading of responses will be performed by assessors blinded to participant identity and treatment assignment.
• Purpose: TREATMENT
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Assistant Professor of Medicine

Model Details: The trial will be designed as a randomized, two-arm, single-blind parallel group study.

Study Record Dates

• First Submitted: December 20, 2023
• First Posted: January 17, 2024
• Study Start: December 28, 2023
• Primary Completion: April 19, 2024
• Study Completion: April 19, 2024
• Last Updated: September 27, 2024

Record last verified: 2024-09

Data Sharing

• IPD Sharing: Will not share

Locations

US (1)