Effect of Artificial Intelligence-Augmented Human Instruction on Surgical Simulation Performance

NCT06273579 | Completed

• 1 other identifier: 2010-270, NEU-09-042-Trial 5
• Study Type: interventional
• Target: 88
• Locations: 1 country
• Sites: 1

Brief Summary

At the Neurosurgical Simulation and Artificial Intelligence Learning Centre, we seek to provide surgical trainees with innovative technologies that allow them to improve their surgical technical skills in risk-free environments, potentially improving patient operative outcomes. The Intelligent Continuous Expertise Monitoring System (ICEMS), a deep learning application that assesses and trains neurosurgical technical skill and provides continuous intraoperative feedback, is one such technology that may improve surgical education. In this randomized controlled trial, medical students from four Quebec universities will be blinded and randomized to one of three groups (one control and two experimental). Group 1 (control) will be provided with verbal AI tutor feedback based on the ICEMS error detection. Group 2 will be tutored by a human instructor who will receive ICEMS error data and deliver verbal instruction using the same words as the ICEMS. Group 3 will be tutored by a human instructor who will be provided with ICEMS data and will then deliver personalized feedback. The aim of this study is to determine how the method of delivery of verbal surgical error instruction influences trainee technical skill acquisition and transfer. Evaluating trainee responses to AI instructor verbal feedback as compared to feedback from human instructors will allow for further development, testing, and optimization of the ICEMS and other AI tutoring systems.

Conditions

Surgical Education

Keywords

Virtual Reality; Artificial Intelligence; Surgical Training; Surgical Education; Surgical Simulation; Neurosurgery

Outcome Measures

Primary Outcomes (2)

• Intelligent Continuous Expertise Monitoring System (ICEMS) expertise score - Technical skill acquisition across practice tasks on NeuroVR simulator

  The ICEMS will continuously evaluate the trainee's performance during each practice task and calculate average expertise scores on a scale of -1.00 (novice) to 1.00 (expert). This will allow us to assess learner technical skill acquisition from the first through sixth repetitions of the practice task.

  1 day of study

• Intelligent Continuous Expertise Monitoring System (ICEMS) expertise score - Technical skill transfer during complex realistic task on NeuroVR simulator

  The ICEMS will continuously evaluate the trainee's performance during the realistic task and calculate an average expertise score on a scale of -1.00 (novice) to 1.00 (expert). This will allow us to assess learner technical skill transfer from the practice tasks to a more complex realistic scenario.

  1 day of study

Secondary Outcomes (2)

• Strength of emotions elicited

  1 day of study

• Levels of cognitive load

  1 day of study

Study Arms (3)

AI tutor instruction group

NO INTERVENTION

31 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal ICEMS feedback when the system detects an error on their performance.

Expert instruction group

EXPERIMENTAL

29 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal feedback from an expert instructor. The expert instructor will deliver this feedback using the same words as the ICEMS.

Behavioral: Expert instruction using AI tutor script

Personalized expert instruction group

EXPERIMENTAL

28 participants allocated. During their second, third, fourth, and fifth repetition of the practice subpial brain tumor resection scenario, participants will receive verbal feedback from an expert instructor. The expert instructor will use their expertise to deliver personalized feedback to the participant.

Behavioral: AI-augmented personalized expert instruction

Interventions

Expert instruction using AI tutor script BEHAVIORAL

Expert instructor assigned to tutor this group will receive error detection data from the ICEMS. They will also be provided with a list of commands that the ICEMS uses. When the system detects an error in a student's performance for a given metric, the instructor must deliver this command using the exact wording provided by the ICEMS.

Expert instruction group

AI-augmented personalized expert instruction BEHAVIORAL

Expert instructor assigned to tutor this group will receive error detection data from the ICEMS. When the system detects an error in a student's performance for a given metric, the instructor will have the freedom to personalize and contextualize feedback without restriction to ICEMS wording.

Personalized expert instruction group

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may not qualify if:

• Prior use of the NeuroVR (CAE Healthcare) simulator.

Sponsors & Collaborators

• McGill University: lead

Study Sites (1)

Neurosurgical Simulation and Artificial Intelligence Learning Centre

Montreal, Quebec, H2X 4B3, Canada

Location

Related Publications (8)

• Yilmaz R, Bakhaidar M, Alsayegh A, Abou Hamdan N, Fazlollahi AM, Tee T, Langleben I, Winkler-Schwartz A, Laroche D, Santaguida C, Del Maestro RF. Real-Time multifaceted artificial intelligence vs In-Person instruction in teaching surgical technical skills: a randomized controlled trial. Sci Rep. 2024 Jul 2;14(1):15130. doi: 10.1038/s41598-024-65716-8.

  PMID: 38956112 BACKGROUND

• Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.

  PMID: 35191972 BACKGROUND

• Yilmaz R, Fazlollahi AM, Winkler-Schwartz A, Wang A, Makhani HH, Alsayegh A, Bakhaidar M, Tran DH, Santaguida C, Del Maestro RF. Effect of Feedback Modality on Simulated Surgical Skills Learning Using Automated Educational Systems- A Four-Arm Randomized Control Trial. J Surg Educ. 2024 Feb;81(2):275-287. doi: 10.1016/j.jsurg.2023.11.001. Epub 2023 Dec 29.

  PMID: 38160107 BACKGROUND

• Fazlollahi AM, Yilmaz R, Winkler-Schwartz A, Mirchi N, Ledwos N, Bakhaidar M, Alsayegh A, Del Maestro RF. AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training. JAMA Netw Open. 2023 Sep 5;6(9):e2334658. doi: 10.1001/jamanetworkopen.2023.34658.

  PMID: 37725373 BACKGROUND

• Mirchi N, Bissonnette V, Yilmaz R, Ledwos N, Winkler-Schwartz A, Del Maestro RF. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLoS One. 2020 Feb 27;15(2):e0229596. doi: 10.1371/journal.pone.0229596. eCollection 2020.

  PMID: 32106247 BACKGROUND

• Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

  PMID: 31373651 BACKGROUND

• Davidovic V, Giglio B, Albeloushi A, Alhaj AK, Alhantoobi M, Saeedi R, Deraiche S, Yilmaz R, Tee T, Fazlollahi AM, Ha M, Uthamacumaran A, Balasubramaniam N, Correa JA, Del Maestro RF. Effect of Artificial Intelligence-Augmented Human Instruction on Feedback Frequency and Surgical Performance During Simulation Training. J Surg Educ. 2025 Nov;82(11):103743. doi: 10.1016/j.jsurg.2025.103743. Epub 2025 Oct 8.

  PMID: 41066879 DERIVED

• Giglio B, Albeloushi A, Alhaj AK, Alhantoobi M, Saeedi R, Davidovic V, Uthamacumaran A, Yilmaz R, Lapointe J, Balasubramaniam N, Tee T, Fazlollahi AM, Correa JA, Del Maestro RF. Artificial Intelligence-Augmented Human Instruction and Surgical Simulation Performance: A Randomized Clinical Trial. JAMA Surg. 2025 Sep 1;160(9):993-1003. doi: 10.1001/jamasurg.2025.2564.

  PMID: 40768205 DERIVED

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: SINGLE
• Who Masked: PARTICIPANT
• Masking Details: Single (Participant) Study participants are blinded to group assignments and study outcomes.
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Director of the Neurosurgical Simulation and Artificial Intelligence Learning Centre

Model Details: Randomized Controlled Trial

Study Record Dates

• First Submitted: February 7, 2024
• First Posted: February 22, 2024
• Study Start: March 9, 2024
• Primary Completion: September 14, 2024
• Study Completion: September 14, 2024
• Last Updated: May 30, 2025

Record last verified: 2025-05

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ICF, CSR, ANALYTIC CODE
• Time Frame: Data will be available for 5 years following the completion of the trial.
• Access Criteria: Researchers who wish to access the data must contact the principal investigator of the trial, Dr. Rolando F. Del Maestro.

Locations

CA (1)