Testing the Efficacy of an Artificial Intelligence Real-Time Coaching SystemSystemSimulatioTraining of Medical Students

NCT05168150 | Completed

• 1 other identifier: 2010-270, NEU-09-042-Trial 3
• Study Type: interventional
• Target: 98
• Locations: 1 country
• Sites: 1

Brief Summary

Background: Trainees learn surgical technical skills through apprenticeship model working closely with surgeons and given increased responsibility in patient cases under expert supervision. However, factors such as surgeons' busy schedule, number of available patient cases, patient safety and lack of objectivity and standardization in training pose strong limitations. Virtual reality surgical simulators integrated with artificial intelligence (AI) systems provide a standardized realistic simulation environment and detailed performance data that allows accurate quantitation of surgical skills and tailored feedback. These platforms make repetitive practice of surgical skills possible in a risk-free environment. The Intelligent Continuous Monitoring System (ICEMS), a deep learning application integrated in NeuroVR simulation platform, was developed to assess surgical performance continuously in 0.2 second intervals and provide coaching and risk detection. Although a predictive validity for assessment module was provided previously, the effectiveness of real-time coaching and risk detection ability with this AI system remains to be explored. The objective of this study is to compare the error-oriented intelligent feedback provided by the ICEMS to in-person expert instruction in surgical simulation training by monitoring the improvement of medical student technical skills on a series of virtual reality tumor resection tasks.

Conditions

Surgical Education

Keywords

Virtual reality; Artificial Intelligence; Coaching

Outcome Measures

Primary Outcomes (2)

• Change in performance

  Performance will be measured using the composite-score assessed by the ICEMS system

  1 Day of Study

• Transfer of learning

  Performance on the complex realistic scenario will be evaluated by the composite-score assessed by the ICEMS system.

  1 Day of Study

Secondary Outcomes (3)

• Objective Structured Assessment of Technical Skills (OSATS) global rating scale

  1 Day of Study

• Differences in strength of emotions elicited

  1 Day of Study

• Difference in Cognitive Load

  1 Day of study

Study Arms (3)

Control Group No-expert mediated post hoc benchmark group

NO INTERVENTION

30 participants. Individuals receive identical introductory information,same time, to perform, same scenarios as other groups. Students receive their scores on 5 performance metrics compared to expert performance benchmarks. Scores are presented in the 5 minute breaks between tasks. Student goal is to be within the benchmark in all five metrics.

Experimental Group - Intelligent Continuous Expertise Monitoring System group

EXPERIMENTAL

30 Participants. Introductory information provided on simulator and scenario. They perform 5 simple practice subpial tumor resections with 5 minutes per trial. On 6th attempt 13 minutes to perform a complex realistic scenario. During first practice task, participants receive no feedback. For the subsequent 4 practice tasks participants will receive real-time auditory feedback instruction by the intelligent system. After each of the 5 attempts, a student takes a 5-minute break. During each of the 5 breaks the participants will be shown the errors they made during the task by the intelligent system regarding five performance metrics monitored. After seeing each error outline, the participant will be shown a video demonstration to learn how to expertly perform on each performance metric. On their 6th attempt they will perform on the realistic scenario without any feedback given.

Behavioral: Experimental: Experimental Group - Intelligent Continuous Expertise Monitoring System group

Experimental Group In-person expert-mediated instruction group

EXPERIMENTAL

30 Participants. Introductory information provided on simulator and scenario. They perform 5 simple practice subpial tumor resections with 5 minutes per trial. On 6th attempt 13 minutes to perform a complex realistic scenario. During first practice task participants receive no feedback. For the subsequent 4 practice tasks participants receive real-time auditory feedback instruction by in-person expert during the task. After each of the 5 tasks, students takes a 5-minute break. During each of the 5 breaks the in-person expert provides feedback to the participant based on their OSATS score assessment during the previous trial. If the expert feels it is appropriate the expert will demonstrate how to do the specific procedure which has been found to be a concern on the simulator themselves so the participant can understand how to improve their performance. On their 6th attempt they will perform on the realistic scenario without any feedback given.

Behavioral: Experimental Group In-person expert-mediated instruction group

Interventions

Experimental: Experimental Group - Intelligent Continuous Expertise Monitoring System group BEHAVIORAL

During each of the practice task they will receive real-time auditory feedback instructed by the intelligent system. After each attempt, a student takes a 5-minute break. They will be shown the errors they made during the task regarding five performance metrics. After seeing each error, they will be shown video demonstration to learn how to expertly perform at each performance metrics. On their 6th attempt they will perform on the realistic scenario without any feedback given.

Experimental Group - Intelligent Continuous Expertise Monitoring System group

Experimental Group In-person expert-mediated instruction group BEHAVIORAL

During each of the practice task, students will receive verbal feedback from the expert instructor present in the room. After each task, experts will summarize their performance and outline the errors the student made. Based on the student's performance, expert will demonstrate how to expertly perform the task in the simulation, and how to improve their performance in the next attempt. Students will perform the 6th attempt on the realistic scenario without any instruction given.

Experimental Group In-person expert-mediated instruction group

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may not qualify if:

• Participation in previous trials involving the NeuroVR (CAE Healthcare) simulator.

Sponsors & Collaborators

• McGill University: lead

Study Sites (1)

Montreal Neurological Institute and Hospital

Montreal, Quebec, H3A 2B4, Canada

Location

Related Publications (4)

• Delorme S, Laroche D, DiRaddo R, Del Maestro RF. NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Neurosurgery. 2012 Sep;71(1 Suppl Operative):32-42. doi: 10.1227/NEU.0b013e318249c744.

  PMID: 22233921 BACKGROUND

• Brightwell A, Grant J. Competency-based training: who benefits? Postgrad Med J. 2013 Feb;89(1048):107-10. doi: 10.1136/postgradmedj-2012-130881. Epub 2012 Sep 27.

  PMID: 23019588 BACKGROUND

• Chan J, Pangal DJ, Cardinal T, Kugener G, Zhu Y, Roshannai A, Markarian N, Sinha A, Anandkumar A, Hung A, Zada G, Donoho DA. A systematic review of virtual reality for the assessment of technical skills in neurosurgery. Neurosurg Focus. 2021 Aug;51(2):E15. doi: 10.3171/2021.5.FOCUS21210.

  PMID: 34333472 BACKGROUND

• Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

  PMID: 31373651 BACKGROUND

Study Officials

• Rolando Del Maestro, MD

  McGill

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: DOUBLE
• Who Masked: PARTICIPANT, OUTCOMES ASSESSOR
• Masking Details: Double (Participant and Expert Rater) Participants do not know the performance metrics used in calculation of their final composite-score, only that they will be learning and practicing technical skills used in neurosurgery while receiving feedback from an instructor or an intelligent system, in subpial tumor resection procedures. Experts do not know to which group the video performance they are rating belongs, for the OSATS rating.
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Director, Neurosurgical Simulation and Artificial Intelligence Learning Centre

Model Details: Randomized Control trial

Study Record Dates

• First Submitted: December 20, 2021
• First Posted: December 23, 2021
• Study Start: January 5, 2022
• Primary Completion: May 3, 2022
• Study Completion: May 3, 2022
• Last Updated: August 8, 2022

Record last verified: 2022-08

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ICF, CSR, ANALYTIC CODE
• Time Frame: Data will be available for 5 years after completion of trial.
• Access Criteria: Researchers wanting access to the data will need to contact the principal investigator of the trial. Dr. Rolando Del Maestro

Locations

CA (1)