Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake

NCT05009251 | Completed Results DMC

• 2 other identifiers: 2021-0483P30AG034532
• Study Type: interventional
• Target: 45,061
• Locations: 1 country
• Sites: 1

Brief Summary

The study team previously demonstrated that patients are more likely to receive flu vaccine after learning that they are at high risk for flu complications. Building on this past work, the present study will explore whether providing reasons that patients are considered high risk for flu complications (a) further increases the likelihood they will receive flu vaccine and (b) decreases the likelihood that they receive diagnoses of flu and/or flu-like symptoms in the ensuing flu season. It will also examine whether informing patients that their high-risk status was determined by analyzing their medical records or by an artificial intelligence (AI) / machine-learning (ML) algorithm analyzing their medical records will affect the likelihood of receiving the flu vaccine or diagnoses of flu and/or flu-like symptoms.

Conditions

Influenza; Vaccination; Health Promotion; Health Behavior; Risk Reduction

Keywords

Flu Vaccine; Choice Architecture; Machine Learning; Perceived Credibility; Personalized Risk Factors; Flu Complications

Outcome Measures

Primary Outcomes (1)

• Flu Vaccination at 2 Weeks After Final Outreach Date

  Received flu vaccination

  Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Secondary Outcomes (4)

• Flu Vaccination at 9 Weeks After Final Outreach Date

  Within 9 weeks of the final outreach date, 106 days (15.14 weeks) after the study start

• Flu Diagnosis

  8 months (between September 9, 2021 and April 30, 2022)

• Flu Complications

  11 months (between September 9, 2021 and July 31, 2022)

• Healthcare Utilization

  11 months (between September 9, 2021 and July 31, 2022)

Other Outcomes (3)

• Flu Vaccination at 2 Weeks After Final Outreach Date by Gender

  Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

• Flu Vaccination at 2 Weeks After Final Outreach Date by Race

  Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

• Flu Vaccination at 2 Weeks After Final Outreach Date by Ethnicity

  Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Study Arms (5)

No-Contact Control

NO INTERVENTION

Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.

Reminder Control

EXPERIMENTAL

Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.

Behavioral: Reminder

High Risk Only

EXPERIMENTAL

Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.

Behavioral: Reminder; Behavioral: Risk reduction

High Risk with Explanation Based on Medical Records

EXPERIMENTAL

Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.

Behavioral: Reminder; Behavioral: Risk reduction; Behavioral: Medical records-based recommendation

High Risk with Explanation Based on Algorithm

EXPERIMENTAL

Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.

Behavioral: Reminder; Behavioral: Risk reduction; Behavioral: Medical records-based recommendation; Behavioral: Algorithm-based recommendation

Interventions

Reminder BEHAVIORAL

Mailed letter, short message service (SMS) text, and/or patient portal message

High Risk Only; High Risk with Explanation Based on Algorithm; High Risk with Explanation Based on Medical Records; Reminder Control

Risk reduction BEHAVIORAL

Mailed letter, SMS, and/or patient portal message

High Risk Only; High Risk with Explanation Based on Algorithm; High Risk with Explanation Based on Medical Records

Medical records-based recommendation BEHAVIORAL

Mailed letter, SMS, and/or patient portal message

Also known as: Credibility

High Risk with Explanation Based on Algorithm; High Risk with Explanation Based on Medical Records

Algorithm-based recommendation BEHAVIORAL

Mailed letter, SMS, and/or patient portal message

Also known as: Credibility

High Risk with Explanation Based on Algorithm

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: No
• Age Groups: Adult (18-64), Older Adult (65+)

You may qualify if:

• Aged 18 or older
• Current Geisinger patient at the time of study
• Falls in the top 10% of patients at highest risk, as identified by the flu-complication risk scores of machine learning algorithm (which operates on coded EHR data)

You may not qualify if:

• Has contraindications for flu vaccination
• Has opted out of receiving communications from Geisinger via all of the modalities being tested

Sponsors & Collaborators

• National Bureau of Economic Research, Inc.: lead
• Geisinger Clinic: collaborator
• Massachusetts Institute of Technology: collaborator
• National Institute on Aging (NIA): collaborator

Study Sites (1)

Geisinger Clinic

Danville, Pennsylvania, 17822, United States

Location

MeSH Terms

Conditions

Influenza, Human; Health Behavior; Risk Reduction Behavior

Interventions

Numbers Needed To Treat

Condition Hierarchy (Ancestors)

Respiratory Tract Infections; Infections; Orthomyxoviridae Infections; RNA Virus Infections; Virus Diseases; Respiratory Tract Diseases; Behavior

Intervention Hierarchy (Ancestors)

Sample Size; Research Design; Methods; Investigative Techniques

Results Point of Contact

• Title: Gail Rosenbaum
• Organization: Geisinger

grosenbaum@geisinger.edu

800-275-6401

Study Officials

• Michelle N Meyer, PhD JD

  Geisinger Clinic

  PRINCIPAL INVESTIGATOR

• Christopher F Chabris, PhD

  Geisinger Clinic

  PRINCIPAL INVESTIGATOR

Publication Agreements

• PI is Sponsor Employee: No
• Restrictive Agreement: No

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: SINGLE
• Who Masked: CARE PROVIDER
• Masking Details: Providers who prescribe vaccination and diagnose conditions will not be randomized to study arms or informed of patient assignment. Although patients will not be explicitly informed which arm they have been randomized to, they will be aware of the messages they receive.
• Purpose: PREVENTION
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: SPONSOR

Model Details: Patients from the high-risk sample will be randomly assigned to one of five arms: * No-Contact Control Arm * Reminder Control Arm * High Risk Only Arm * High Risk with Explanation Based on Medical Records Arm * High Risk with Explanation Based on Algorithm Arm

Study Record Dates

• First Submitted: August 4, 2021
• First Posted: August 17, 2021
• Study Start: September 9, 2021
• Primary Completion: November 5, 2021
• Study Completion: July 31, 2022
• Last Updated: January 3, 2025
• Results First Posted: January 12, 2023

Record last verified: 2024-12

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, CSR, ANALYTIC CODE
• Time Frame: By the paper's online publication date. Data will remain available for as long as the Open Science Framework hosts it.
• Access Criteria: The data and code are available in the below repository in the folder ClinicalTrials.gov data/Study 2.

Locations

US (1)