Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through a Local Large Language Model

NCT06624605 | Completed

• 1 other identifier: 24-000550
• Study Type: interventional
• Target: 851
• Locations: 1 country
• Sites: 1

Brief Summary

This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers\' comprehension and satisfaction with the notes and ophthalmologists\' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.

Conditions

Communication; Artificial Intelligence (AI); Artificial Intelligence Technology; Interdisciplinary Communication

Outcome Measures

Primary Outcomes (1)

• Non-Ophthalmologist Note Comprehension and Satisfaction

  Assessed via survey responses evaluating understanding of the patient\'s ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary. Degree of understanding was assessed on the following Likert scale: Not at all Neutral Moderately A great deal Satisfaction was graded on the following Likert scale: Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied Preference between note types was assessed on the following Likert scale: Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal

  From enrollment to 8 weeks after enrollment

Secondary Outcomes (6)

• Ophthalmologist Evaluation of Summary Accuracy and Time burden

  Assessed at a single time point <24 hr after enrollment

• Semantic and Linguistic Quality of Plain Language Summaries

  Assessed at a single time point <24 hr following enrollment

• Semantic and Linguistic Quality of Plain Language Summaries

  Assessed at a single time point <24 hr following enrollment

• Semantic and Linguistic Quality of Plain Language Summaries

  Assessed at a single time point <24 hr following enrollment

• Semantic and Linguistic Quality of Plain Language Summaries

  Assessed at a single time point <24 hr following enrollment

Other Outcomes (1)

• Presence of Errors, Type of Errors, and Potential Harm of Errors

  Assessed at a single time point <24 hr following enrollment and again at a single time point 8 weeks after enrollment

Study Arms (2)

Control

NO INTERVENTION

Participants receive standard ophthalmology notes without any additions.

Intervention

EXPERIMENTAL

Participants receive standard ophthalmology notes with an appended LLM-generated plain language summary, reviewed and edited by the ophthalmologist. The plain language summary is clearly marked as generated as part of a Quality Improvement initiative.

Other: Large Language Model-generated Plain Language Summary of Ophthalmology notes

Interventions

Large Language Model-generated Plain Language Summary of Ophthalmology notes OTHER

Prospective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes.

Intervention

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: No
• Age Groups: Adult (18-64), Older Adult (65+)

You may qualify if:

• Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.

You may not qualify if:

• No in-network non-Ophthalmologist care providers in the medical record
• Procedure-only visits containing only a procedure note

Sponsors & Collaborators

• John J Chen: lead

Study Sites (1)

Mayo Clinic

Rochester, Minnesota, 55905, United States

Location

MeSH Terms

Conditions

Communication

Condition Hierarchy (Ancestors)

Behavior

Study Officials

• John J Chen, MD, PhD

  Mayo Clinic

  STUDY DIRECTOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: NONE
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: SPONSOR INVESTIGATOR
• PI Title: Principal Investigator

Model Details: Randomized study assessing the efficacy of Large Language Model-generated Plain Language Summaries of Ophthalmology notes in increasing non-ophthalmologist note understanding. Notes were randomized to intervention (Summary + Original Note) or control (Original note only) arms. Non-ophthalmology providers were sent surveys assessing note comprehension for both control and treatment arm notes, and Ophthalmologists were sent surveys for intervention arm notes regarding Summary accuracy and other attributes. Conducted as a Quality Improvement initiative.

Study Record Dates

• First Submitted: September 30, 2024
• First Posted: October 3, 2024
• Study Start: February 1, 2024
• Primary Completion: May 31, 2024
• Study Completion: May 31, 2024
• Last Updated: October 3, 2024

Record last verified: 2024-10

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, CSR, ANALYTIC CODE
• Time Frame: Beginning after publication with no end date
• Access Criteria: The underlying data supporting the findings of this study can be made available to clinical investigators and researchers upon request. Written requests for data sharing including an analysis plan will be required before approval. These requests will be individually assessed in consultation with the study team leads and co-investigators as appropriate. If other investigators are interested in performing additional analyses, these requests can be made to the corresponding author (PDT) and analyses will be performed in collaboration with the Mayo Clinic. In all cases, any data and materials to be shared will be released via a Material Transfer Agreement. Individual-level data will be available and data sharing will ensure that the rights and privacy of individuals participating in the research always remains protected. The anticipated time frame to respond to initial data requests is 1 month.

Locations

US (1)