Encouraging Flu Vaccination Among High-Risk Patients Identified by ML

NCT04323137 | Completed Results

• 2 other identifiers: 2020-02905P30AG034532-12
• Study Type: interventional
• Target: 117,649
• Locations: 1 country
• Sites: 1

Brief Summary

The purpose of the current study is to test different interventions to determine the most effective way to promote flu vaccine uptake in a high-risk population identified by an "artificial intelligence" (AI) or machine learning (ML) algorithm. The specific aims are:

1. 1.Evaluate the effect on flu vaccination rates of informing health-system patients who are identified by an ML analysis of EHR data to be at high risk for flu complications that they are at high risk with either (a) no additional explanation, (b) an explanation that this determination comes from an analysis of their medical records, and (c) the additional explanation that an AI or ML algorithm made this determination.
2. 2.Evaluate the effects of the same three interventions on diagnoses of flu in the same patients.

Conditions

Influenza; Vaccination; Health Promotion; Health Behavior; Risk Reduction

Keywords

Flu Vaccine; Choice Architecture; Machine Learning; Perceived Credibility

Outcome Measures

Primary Outcomes (3)

• Flu Vaccination Rate

  Patient received a flu vaccination

  Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration

• Flu Vaccination Rate by Risk Level

  Patient received a flu vaccination Note: For patients who received risk communications, those in the top 3% were always told they were in the top 3% of risk. Those in the top 4-10% of risk were randomized to be told that they were in the top 10% of risk or high risk. Control patients in the top 3% and top 4-10% of risk were allocated to the top 3% and randomized to either top 10% or high risk groups, respectively, at the same time as those in the patient contact groups, even though these control patients were not contacted.

  Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration

• High Confidence Flu Diagnosis Rate

  Patient received a flu diagnosis via a positive PCR/antigen/molecular test

  Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration

Secondary Outcomes (12)

• "Likely Flu" Diagnosis Rate

  Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration

• Flu Complications Rate

  Through 3 months after the end of the flu season (August 31st 2021), approximately 12 months assessment duration

• Change in ER Visits From Pre- to Post-intervention

  Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2)

• Change in Hospitalizations From Pre- to Post-intervention

  Within 12 months pre-intervention (Time 1) and within 12 months post-intervention (Time 2)

• Flu Vaccination Among Fellow Household Members

  Through the the end of the flu season (May 31st 2021), approximately 9 months assessment duration

Study Arms (6)

Control

NO INTERVENTION

This group receives no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.

High risk only

EXPERIMENTAL

This group receives messages telling them they have been identified to be at high risk for flu complications without specifying how or why the health system believes this to be the case.

Behavioral: Risk reduction

High risk based on medical records

EXPERIMENTAL

This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records.

Behavioral: Risk reduction; Behavioral: Medical records-based recommendation

High risk based on algorithm

EXPERIMENTAL

This group receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by an AI/ML system.

Behavioral: Risk reduction; Behavioral: Medical records-based recommendation; Behavioral: Algorithm-based recommendation

Sub-threshold patients

NO INTERVENTION

Patients in this group is in the top 11-20% of risk for flu and complications, slightly lower risk than those included in the intervention, who are in the top 10% of risk for flu and complications. This group of patients does not receive an intervention, but are monitored for flu shots as a comparison to target patients.

Household members

NO INTERVENTION

This group of patients share an address with target high-risk patients (in arms 1-4). This group does not receive an intervention but is monitored for spillover effects of the intervention.

Interventions

Risk reduction BEHAVIORAL

Mailed letter, SMS, and/or patient portal message

High risk based on algorithm; High risk based on medical records; High risk only

Medical records-based recommendation BEHAVIORAL

Mailed letter, SMS, and/or patient portal message

Also known as: Credibility

High risk based on algorithm; High risk based on medical records

Algorithm-based recommendation BEHAVIORAL

Mailed letter, SMS, and/or patient portal message

Also known as: Credibility

High risk based on algorithm

Eligibility Criteria

• Age: 17 Years+
• Sex: all
• Healthy Volunteers: No
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)

You may qualify if:

• Current Geisinger patient at the time of study
• Falls in the top 10% of patients at highest risk, as identified by the flu-complication risk scores of Medial's machine learning algorithm (which operates on coded EHR data)

You may not qualify if:

• Has contraindications for flu vaccination
• Has opted out of receiving communications from Geisinger via all of the modalities being tested

Sponsors & Collaborators

• Geisinger Clinic: lead
• National Institute on Aging (NIA): collaborator

Study Sites (1)

Geisinger

Danville, Pennsylvania, 17822, United States

Location

Related Publications (13)

• Bigman YE, Gray K. People are averse to machines making moral decisions. Cognition. 2018 Dec;181:21-34. doi: 10.1016/j.cognition.2018.08.003. Epub 2018 Aug 11.

  PMID: 30107256 BACKGROUND

• Dietvorst BJ, Simmons JP, Massey C. Algorithm aversion: people erroneously avoid algorithms after seeing them err. J Exp Psychol Gen. 2015 Feb;144(1):114-26. doi: 10.1037/xge0000033. Epub 2014 Nov 17.

  PMID: 25401381 BACKGROUND

• Goshen R, Choman E, Ran A, Muller E, Kariv R, Chodick G, Ash N, Narod S, Shalev V. Computer-Assisted Flagging of Individuals at High Risk of Colorectal Cancer in a Large Health Maintenance Organization Using the ColonFlag Test. JCO Clin Cancer Inform. 2018 Dec;2:1-8. doi: 10.1200/CCI.17.00130.

  PMID: 30652563 BACKGROUND

• Logg, J.M., Minson, J.A., & Moore, D.A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103. https://doi.org/10.1016/j.obhdp.2018.12.005

  BACKGROUND

• Shen N, Bernier T, Sequeira L, Strauss J, Silver MP, Carter-Langford A, Wiljer D. Understanding the patient privacy perspective on health information exchange: A systematic review. Int J Med Inform. 2019 May;125:1-12. doi: 10.1016/j.ijmedinf.2019.01.014. Epub 2019 Feb 1.

  PMID: 30914173 BACKGROUND

• Rothberg MB, Haessler SD. Complications of seasonal and pandemic influenza. Crit Care Med. 2010 Apr;38(4 Suppl):e91-7. doi: 10.1097/CCM.0b013e3181c92eeb.

  PMID: 19935413 BACKGROUND

• Tokars JI, Olsen SJ, Reed C. Seasonal Incidence of Symptomatic Influenza in the United States. Clin Infect Dis. 2018 May 2;66(10):1511-1518. doi: 10.1093/cid/cix1060.

  PMID: 29206909 BACKGROUND

• Turner D, Wailoo A, Nicholson K, Cooper N, Sutton A, Abrams K. Systematic review and economic decision modelling for the prevention and treatment of influenza A and B. Health Technol Assess. 2003;7(35):iii-iv, xi-xiii, 1-170. doi: 10.3310/hta7350.

  PMID: 14609480 BACKGROUND

• WHO Guidelines for Pharmacological Management of Pandemic Influenza A(H1N1) 2009 and Other Influenza Viruses. Geneva: World Health Organization; 2010 Feb. Available from http://www.ncbi.nlm.nih.gov/books/NBK138515/

  PMID: 23741777 BACKGROUND

• Zack CJ, Senecal C, Kinar Y, Metzger Y, Bar-Sinai Y, Widmer RJ, Lennon R, Singh M, Bell MR, Lerman A, Gulati R. Leveraging Machine Learning Techniques to Forecast Patient Prognosis After Percutaneous Coronary Intervention. JACC Cardiovasc Interv. 2019 Jul 22;12(14):1304-1311. doi: 10.1016/j.jcin.2019.02.035. Epub 2019 Jun 26.

  PMID: 31255564 BACKGROUND

• Centers for Disease Control and Prevention. (2020). Disease Burden of Influenza. https://www.cdc.gov/flu/ about/burden/index.html (Jan 10).

  BACKGROUND

• Centers for Disease Control and Prevention. (2019a). Who Needs a Flu Vaccine and When. https://www.cdc.gov/flu/prevent/vaccinations.htm (Oct 11).

  BACKGROUND

• Centers for Disease Control and Prevention. (2019b). Flu Vaccination Coverage, United States, 2018-19 Influenza Season. https://www.cdc .gov/flu/fluvaxview/coverage-1819estimates.htm

  BACKGROUND

MeSH Terms

Conditions

Influenza, Human; Health Behavior; Risk Reduction Behavior

Interventions

Numbers Needed To Treat

Condition Hierarchy (Ancestors)

Respiratory Tract Infections; Infections; Orthomyxoviridae Infections; RNA Virus Infections; Virus Diseases; Respiratory Tract Diseases; Behavior

Intervention Hierarchy (Ancestors)

Sample Size; Research Design; Methods; Investigative Techniques

Limitations and Caveats

One primary and multiple secondary outcomes involved analysis of flu diagnoses and complications. However, flu cases and other related outcomes in the 2020-2021 flu season were too low to detect any difference in these more distal outcomes, so results are limited to flu vaccination outcomes.

Results Point of Contact

• Title: Amir Goren, PhD
• Organization: Geisinger Health

agoren@geisinger.edu

570-214-4395

Study Officials

• Christopher Chabris, PhD

  Geisinger Clinic

  PRINCIPAL INVESTIGATOR

Publication Agreements

• PI is Sponsor Employee: Yes
• Restrictive Agreement: No

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: DOUBLE
• Who Masked: PARTICIPANT, CARE PROVIDER
• Masking Details: Participants (i.e., patients) will not be informed specifically of their assignment to different arms throughout the study. Providers who prescribe vaccination and diagnose conditions will not be randomized to study arms or informed of patient assignment.
• Purpose: PREVENTION
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Faculty Co-Director, Behavioral Insights Team

Model Details: Patients from the high-risk sample will be randomly assigned to one of 4 groups: 1. Control: group that receives no additional pro-vaccination intervention beyond Geisinger's normal efforts. 2. High Risk Only: group that receives messages telling them they have been identified to be at high risk for flu complications without specifying how/why Geisinger believes this to be the case 3. High Risk Based on Medical Records: group that receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records 4. High Risk Based on Algorithm: group that receives messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by AI/ML Two additional groups will be monitored for outcome data: 5. Sub-threshold patients: patients who are in the top 11-20% of risk 6. Household members: patients who share an address with target patients

Study Record Dates

• First Submitted: March 19, 2020
• First Posted: March 26, 2020
• Study Start: September 21, 2020
• Primary Completion: May 31, 2021
• Study Completion: September 21, 2021
• Last Updated: December 30, 2024
• Results First Posted: November 7, 2022

Record last verified: 2024-12

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ANALYTIC CODE
• Time Frame: The data will become available after publication of study results in a scientific journal and will be available as long as the Open Science Framework hosts the data.
• Access Criteria: The data and code are available in the OSF repository at the url below in the folder ClinicalTrials.gov data/Study 1.

Locations

US (1)