AI-Based Fidelity Feedback to Enhance CBT

NCT05340738 | Completed

• 1 other identifier: R42MH128101
• Study Type: interventional
• Target: 449
• Locations: 1 country
• Sites: 1

Brief Summary

This study is being conducted together by researchers at the University of Pennsylvania and Lyssn.io, Inc., ("Lyssn"), a technology start-up developing digital tools to support evidence-based psychotherapies (EBPs) for mental health disorders and addiction. This study will implement a technology to assess and enhance the quality of EBPs like Cognitive Behavioral Therapy (CBT) that includes a user interface geared to clinical, supervision, and administrative workflows and needs, and then assess this technology for effectiveness in comparison to usual care. There is a tremendous global burden of mental illness: Over 50 million American adults have a diagnosable mental health disorder, and major depression on its own is the leading cause of disability worldwide. In the face of this burden, clinical research has documented a variety of effective EBPs (e.g. CBT), and these psychotherapies are utilized on a massive scale. Systems have invested over $2 billion in training providers in specific EBPs. Once trained, however, therapists' adherence to the EBP, also called fidelity, is both crucial for effectiveness and difficult to assess. There is no scalable method to assess the fidelity and quality of EBPs in community practice settings. This is a foundational problem for healthcare systems. Advances in speech processing and machine learning make technology a promising solution to this problem. The use of technology - instead of humans - to evaluate EBPs means that objective, performance-based feedback can be provided quickly, efficiently, cost-effectively, and without human error. If successful, the present research will be among the first examples of a method for building, monitoring, and assessing the quality of therapy that can scale up to large, real-world healthcare settings. In this study, the investigators will implement an existing, fully-functional prototype (LyssnCBT) that includes a user interface geared to community mental health (CMH) clinical, supervision, and administrative workflows and needs, and then assess for effectiveness of psychotherapy supported by LyssnCBT in comparison to usual care. This study will implement LyssnCBT in 5 community mental health agencies, beginning with a single-arm pilot field trial to identify and address any specific barriers to implementing the tool in a community mental health context. The study team will then conduct a larger study in community mental health agencies comparing LyssnCBT to services as usual.

Conditions

Cognitive Behavioral Therapy; Therapy

Keywords

depression; cognitive behavioral therapy; therapy; evidence-based psychotherapy; training; community mental health; CBT fidelity; anxiety

Outcome Measures

Primary Outcomes (8)

• Change in therapist CBT fidelity as assessed by CTRS scores

  CBT Fidelity will be assessed by AI-generated Cognitive Therapy Rating Scale (CTRS) scores for every recorded therapy session, which will be recorded via the Lyssn platform during both SAU and LyssnCBT phases of the study. CBT sessions are scored on different session components (agenda, feedback, understanding, etc.) with the CTRS on a scale of 0-6, from "poor" to "excellent", where higher scores indicate a greater degree of CBT fidelity. CTRS deviation scores will be created.Mean CTRS scores during SAU will be estimated for each therapist and for each CTRS item. These means will be subtracted from each corresponding CTRS score during the LyssnCBT phase.

  Phase 2; Immediately after every recorded therapy session

• Acceptability of LyssnCBT tools (Acceptability of Intervention Measure; AIM)

  A brief, 4-item measure of a mental health professional's perception of how acceptable an innovation would be for use in their context. Each statement about the acceptability of the innovation is rated on a 5-point ordinal scale ranging from "completely disagree" to "completely agree," where higher scores indicate higher levels of acceptability. This measure has demonstrated good test-retest reliability and sensitivity to change among community mental health providers.

  After three months of engagement with the LyssnCBT tools

• Appropriateness of LyssnCBT tools (Intervention Appropriateness Measure; IAM)

  A brief, 4-item measure of a mental health professional's perception of how appropriate an innovation would be for use in their context. Each statement about the appropriateness of the innovation is rated on a 5-point ordinal scale ranging from "completely disagree" to "completely agree," where higher scores indicate higher levels of appropriateness. This measure has demonstrated good test-retest reliability and sensitivity to change among community mental health providers.

  Phase 2; Three months after the site moves from SAU to LyssnCBT

• Feasibility of LyssnCBT tools (Feasibility of Intervention Measure; FIM)

  A brief, 4-item measure of a mental health professional's perception of how feasible an innovation would be for use in their context. Each statement about the feasibility of the innovation is rated on a 5-point ordinal scale ranging from "completely disagree" to "completely agree," where higher scores indicate higher levels of feasibility. This measure has demonstrated good test-retest reliability and sensitivity to change among community mental health providers.

  Phase 2; Three months after the site moves from SAU to LyssnCBT

• Usability of LyssnCBT tools (System Usability Scale; SUS)

  A brief, 10-item questionnaire to evaluate an end-user's perceptions of an innovation's usability. Each statement is rated on a 5-point ordinal scale ranging from "strongly disagree" to "strongly agree," with every other question reverse-coded. Scores are converted to a common scale, added together and then multiplied by 2.5 to convert the original scores to a scale of 0-100. "Average" usability is defined as a total score \>68

  Phase 2; Three months after the site moves from SAU to LyssnCBT

• Change in client depression symptoms (Patient Health Questionnaire; PHQ-9)

  The PHQ-9 (Patient Health Questionnaire) is a brief, widely used depression inventory assessing frequency of symptoms with 9 total items rated on a likert scale of 0-3. Higher scores indicate a greater frequency of symptoms of depression and presence of a clinically significant depressive disorder. This measure has demonstrated good test-retest reliability and can be completed within 2-3 minutes. This measure will be collected at every therapy session.

  Immediately before weekly therapy sessions for the duration of treatment or through study completion, a maximum of 18 months

• Change in client anxiety symptoms (General Anxiety Disorder; GAD-7)

  The GAD-7 (General Anxiety Disorder) is a brief, widely used anxiety inventory assessing frequency of symptoms with 7 total items rated on a likert scale of 0-3. Higher scores indicate a greater frequency of symptoms of anxiety and presence of a clinically significant anxiety disorder. This measure has demonstrated good test-retest reliability and can be completed within 2-3 minutes. This measure will be collected at every therapy session.

  Immediately before weekly therapy sessions for the duration of treatment or through study completion, a maximum of 18 months

• Client drop-out/premature termination from therapeutic engagement

  Dropout is collected via a brief monthly survey sent out to participating therapists which will ask, for each participating client, whether the client has been discharged and if that discharge was planned.

  Immediately before weekly therapy sessions for the duration of treatment or through study completion, a maximum of 18 months

Study Arms (2)

LyssnCBT

ACTIVE COMPARATOR

Therapists will use the LyssnCBT tool with clients for recording and session-sharing functionalities. Therapists and supervisors will also have access to LyssnCBT features like speech-to-text transcription, annotation tools, and AI-generated metrics.

Other: LyssnCBT

SAU (services-as-usual)

NO INTERVENTION

Therapists will use the LyssnCBT tool with clients for recording and session-sharing functionalities. No other LyssnCBT features will be available for therapist or supervisor review.

Interventions

LyssnCBT OTHER

LyssnCBT is a technology that allows therapists and supervisors access to tools that assist with assessing CBT session fidelity, including speech-to-text transcription, annotation tools, and AI-generated metrics.

LyssnCBT

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may qualify if:

• Therapists
• Able to participate in therapy sessions conducted in English
• Employed at a Philadelphia CMH treatment center that allows the recruitment and participation of therapists in research-related activities
• Willing to allow their session recordings to be used for research purposes
• Computer and internet access
• Supervisors
• Oversee participating therapists
• Computer and internet access
• Clients
• Able to participate in therapy sessions conducted in English
• Willing to allow the team to collect data and use their session recordings for research purposes

You may not qualify if:

• Unwilling to allow the research team to access their therapy session recordings

Sponsors & Collaborators

• University of Pennsylvania: lead
• Lyssn.io, Inc.: collaborator
• National Institute of Mental Health (NIMH): collaborator

Study Sites (1)

The Penn Collaborative for CBT and Implementation Science

Philadelphia, Pennsylvania, 19104, United States

Location

Related Publications (1)

• Creed TA, Salama L, Slevin R, Tanana M, Imel Z, Narayanan S, Atkins DC. Enhancing the quality of cognitive behavioral therapy in community mental health through artificial intelligence generated fidelity feedback (Project AFFECT): a study protocol. BMC Health Serv Res. 2022 Sep 20;22(1):1177. doi: 10.1186/s12913-022-08519-9.

  PMID: 36127689 DERIVED

MeSH Terms

Conditions

Depression; Anxiety Disorders

Condition Hierarchy (Ancestors)

Behavioral Symptoms; Behavior; Mental Disorders

Study Officials

• Torrey A Creed, PhD

  Director, The Penn Collaborative for CBT and Implementation Science

  PRINCIPAL INVESTIGATOR

• David Atkins, PhD

  CEO, Lyssn

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: NONE
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: CROSSOVER
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Assistant Professor of Psychology in Psychiatry

Model Details: Type 2 hybrid implementation-effectiveness, randomized stepped-wedge study comparing LyssnCBT-supported psychotherapy to services as usual (SAU)

Study Record Dates

• First Submitted: February 18, 2022
• First Posted: April 22, 2022
• Study Start: March 9, 2023
• Primary Completion: October 31, 2025
• Study Completion: January 31, 2026
• Last Updated: March 24, 2026

Record last verified: 2026-03

Data Sharing

• IPD Sharing: Will not share

Locations

US (1)