AI-generated Feedback in Social Robotic Virtual Patients

NCT07277829 | Completed

• 1 other identifier: 2024-05876-02
• Study Type: interventional
• Target: 115
• Locations: 1 country
• Sites: 1

Brief Summary

The goal of this quasi-experimental educational study is to learn whether AI-generated post-consultation feedback in social robotic virtual patient interactions improves medical students' clinical performance in medical history-taking. The main question it aims to answer is: Can AI-generated feedback integrated in an AI-enhanced social robotic virtual patient platform improve medical students' clinical performance in medical history taking? Researchers will compare results from standardised examinations following the structure of an objective structured clinical examination (OSCE), of medical students performing virtual patient interactions with AI-generated post consultation feedback compared to medical students who have not received AI-generated feedback. Participants will perform five virtual patient cases in rheumatology using an established virtual patient platform: the Social AI-enhanced Robotic Interface (SARI). After completion of each case, students participate in follow-up seminars with consultant rheumatologists to discuss the cases. After completion of all nine cases, students take part in a OSCE based examination to evaluate medical-history taking.

Conditions

Virtual Patient; Social Robots; Artificial Intelligence (AI); Large Language Model; Feedback; Medical History Taking

Keywords

AI; Virtual Patient; Medical History Taking; Social Robotics; Feedback; Large Language Model

Outcome Measures

Primary Outcomes (1)

• Total OSCE score

  Immediately after completing the virtual outpatient clinic, participating students underwent an OSCE-like evaluation designed to assess performance in medical history-taking and clinical reasoning. The eight-minute evaluation involved interaction with a standardised patient. Performance was assessed by a consultant rheumatologist using a validated rubric, with the assessor blinded to whether students had received feedback. The assessment used a ten-point scale across five domains: communication at consultation start (0-3 points), generic medical history (0-3.5 points), targeted medical history (0-1.5 points), diagnostics and management reasoning (0-1 point), and communication at consultation end (0-1 point).

  Enrollment and evaluation during the same week during a period of 1.5 working days. Evaluation was done following completion of all virtual patient cases at the end of Day 2 of the Virtual Outpatient Clinic.

Secondary Outcomes (2)

• Pass rate comparisons

  Enrollment and evaluation during the same week during a period of 1.5 working days. Evaluation was done following completion of all virtual patient cases at the end of Day 2 of the Virtual Outpatient Clinic.

• Individual OSCE domain comparisons

  Enrollment and evaluation during the same week during a period of 1.5 working days. Evaluation was done following completion of all virtual patient cases at the end of Day 2 of the Virtual Outpatient Clinic.

Study Arms (2)

AI-generated feedback

EXPERIMENTAL

AI-generated post consultation feedback following interaction with the AI-enhanced social robotic virtual patient platform the Social AI-enhanced Robotic Interface (SARI)

Device: AI post consultation feedback

Control

NO INTERVENTION

Interaction with the AI-enhanced social robotic virtual patient platform the Social AI-enhanced Robotic Interface (SARI) with no post consultation feedback.

Interventions

AI post consultation feedback DEVICE

A feedback algorithm which follows a two-stage design was implemented to generate post consultation feedback using large language models (LLMs) from OpenAI. The first stage of the feedback algorithm is an assessment model that evaluates student-VP dialogues using a predefined rubric developed in collaboration with consultant rheumatologists. This assessment model was iteratievly refined and validated prior to the study. The second stage of the feedback algorithm involves generating a feedback output based on the stageone assessment Students received approximately one page of structured written feedback immediately after completing each VP encounter with SARI. This feedback focused on medical history-taking within the context of rheumatology and included constructive comments with examples covering general history-taking, specific symptom enquiries, and systematic assessment of the VPs.

AI-generated feedback

Eligibility Criteria

• Sex: all
• Healthy Volunteers: No
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)

You may qualify if:

• Sixth-semester medical students at Karolinska Institutet.
• Assigned to clinical rotations in rheumatology.
• Participating between January and June 2025.

You may not qualify if:

• \- None.

Sponsors & Collaborators

• Ioannis Parodis: lead
• Region Stockholm: collaborator
• Karolinska Institutet: collaborator

Study Sites (1)

Karolinska University Hospital

Stockholm, Solna, SE-171 76, Sweden

Location

Related Publications (2)

• Borg A, Jobs B, Huss V, Gentline C, Espinosa F, Ruiz M, Edelbring S, Georg C, Skantze G, Parodis I. Enhancing clinical reasoning skills for medical students: a qualitative comparison of LLM-powered social robotic versus computer-based virtual patients within rheumatology. Rheumatol Int. 2024 Dec;44(12):3041-3051. doi: 10.1007/s00296-024-05731-0. Epub 2024 Oct 16.

  PMID: 39412574 RESULT

• Borg A, Georg C, Jobs B, Huss V, Waldenlind K, Ruiz M, Edelbring S, Skantze G, Parodis I. Virtual Patient Simulations Using Social Robotics Combined With Large Language Models for Clinical Reasoning Training in Medical Education: Mixed Methods Study. J Med Internet Res. 2025 Mar 3;27:e63312. doi: 10.2196/63312.

  PMID: 40053778 RESULT

Study Officials

• Ioannis Parodis, MD, PhD

  Karolinska Institutet

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: NON RANDOMIZED
• Masking: DOUBLE
• Who Masked: INVESTIGATOR, OUTCOMES ASSESSOR
• Masking Details: Medical students/participants will know if they are part of intervention or control arm. Outcome assessors will be blinded to participant conditions during OSCE based examinations.
• Purpose: OTHER
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: SPONSOR INVESTIGATOR
• PI Title: MD, PhD

Study Record Dates

• First Submitted: November 20, 2025
• First Posted: December 11, 2025
• Study Start: January 27, 2025
• Primary Completion: June 5, 2025
• Study Completion: June 5, 2025
• Last Updated: December 11, 2025

Record last verified: 2025-11

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP

Locations

SE (1)