Effectiveness of an Artificial Intelligent Tutoring System in Simulation Training

NCT04700384 | Completed

• 1 other identifier: 2010-270, NEU-09-042
• Study Type: interventional
• Target: 70
• Locations: 1 country
• Sites: 1

Brief Summary

Brief Summary: Background: Although surgical experience and technical skill are associated with better patient outcomes, quantitating surgical ability in the operating room is challenging. In surgical education, large datasets generated by high-fidelity virtual reality simulators can be employed by machine learning algorithms to objectively measure trainee performance and competence on expert benchmarks. This allows repetitive practice of surgical skills in safe and risk-free environments with immediate feedback. Our group developed and has a patent pending for an intelligent tutoring system called the Virtual Operative Assistant (VOA). Utilizing an Artificial Intelligence (AI) support vector machine algorithm, the VOA assesses data derived from the NeuroVR (CAE Healthcare) simulator platform and provides individualized audiovisual feedback to improve learner performance during simulated brain tumor resections. The effectiveness of intelligent tutoring systems such as the VOA to the human surgical apprenticeship pedagogy remains to be elucidated. The aim of this study is to compare the effectiveness and educational impact of personalized VOA feedback to expert instruction on medical student's technical skills learning of a virtual reality tumor resection procedure. Specific Aims: 1) To assess if medical students receiving personalized VOA feedback statistically improve their surgical performance when compared to those having (a) no expert instructor feedback or (b) expert instructor-mediated feedback. 2) To outline if different emotions are elicited by the VOA intelligent tutoring system in medical students while performing this achievement task as compared to human instruction

Conditions

Surgical Education

Keywords

Virtual Reality; Simulation

Outcome Measures

Primary Outcomes (2)

• Change in procedural performance .

  Performance in each practice attempt is measured utilizing raw data from the simulator that is used for assessment by previously established AI algorithms on validated metrics.

  Day of Study

• Change in learning

  Performance on the complex realistic scenario is evaluated by expert instructors using the Objective Structured Assessments of technical Skills (OSATS) Visual Rating Scale (weighted at 50%) and the AI assessment algorithms (weighted at 50%) creating a composite performance score.

  Day of Study

Secondary Outcomes (2)

• Difference in the strength of emotions elicited

  Day of Study

• Difference in cognitive load

  Day of Study

Study Arms (3)

Control Group

NO INTERVENTION

Control Group - Baseline Training 25 Participants allocated. Individuals receive introductory information on using the simulator and the scenario. They perform 5 simple subpial tumour resections for practice and have 5 minutes per trial. After each attempt, the student takes a 5-minute break with no assessment or feedback on their performance. On their 6th attempt they have 13 minutes to perform a different realistic scenario.

Experimental Group - Virtual Operative Assistance Training

EXPERIMENTAL

Experimental Group - Virtual Operative Assistance Training 25 participants allocated. Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. In the 5-minutes between attempts, participant receive the Virtual Operative Assistance Training assessment of their performance and audiovisual feedback.

Behavioral: Virtual Operative Assistant Training

Experimental Group - remote-based expert Instructor Training

EXPERIMENTAL

25 participants allocated. Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. Meanwhile, a trained instructor observes the participant's on-screen performance, that is live-streamed, remotely. Instructors are senior neurosurgery residents with extensive experience in performing and assessing this scenario. During the 5-minute feedback session, they chat with the student, discussing the performance and help in setting goals for the next trial.

Behavioral: Remote-Based Expert Instructor Training

Interventions

Virtual Operative Assistant Training BEHAVIORAL

Individuals receive the same basic information, have the same amount of time and perform the same scenarios as the control group. In the 5-minutes between attempts, participant receive the Virtual Operative Assistant assessment of their performance and audiovisual feedback.

Experimental Group - Virtual Operative Assistance Training

Remote-Based Expert Instructor Training BEHAVIORAL

Individuals receive the same information, have the same amount of time and perform the same scenarios as the control group. Meanwhile, a trained instructor observes the participant's on-screen performance, that is live-streamed, remotely. Instructors are senior neurosurgery residents with extensive experience in performing and assessing this scenario. During the 5-minute feedback session, they chat with the student, discussing the performance and help in setting goals for the next trial.

Experimental Group - remote-based expert Instructor Training

Eligibility Criteria

• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Child (0-17), Adult (18-64), Older Adult (65+)

Sponsors & Collaborators

• McGill University: lead

Study Sites (1)

Neurosurgical Simulation and Artificial Intelligence Learning Centre

Montreal, Quebec, H3A 2B4, Canada

Location

Related Publications (9)

• Birkmeyer JD, Finks JF, O'Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ; Michigan Bariatric Surgery Collaborative. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013 Oct 10;369(15):1434-42. doi: 10.1056/NEJMsa1300625.

  PMID: 24106936 BACKGROUND

• Birkmeyer JD, Stukel TA, Siewers AE, Goodney PP, Wennberg DE, Lucas FL. Surgeon volume and operative mortality in the United States. N Engl J Med. 2003 Nov 27;349(22):2117-27. doi: 10.1056/NEJMsa035205.

  PMID: 14645640 BACKGROUND

• Stulberg JJ, Huang R, Kreutzer L, Ban K, Champagne BJ, Steele SR, Johnson JK, Holl JL, Greenberg CC, Bilimoria KY. Association Between Surgeon Technical Skills and Patient Outcomes. JAMA Surg. 2020 Oct 1;155(10):960-968. doi: 10.1001/jamasurg.2020.3007.

  PMID: 32838425 BACKGROUND

• Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

  PMID: 31373651 BACKGROUND

• Mirchi N, Bissonnette V, Yilmaz R, Ledwos N, Winkler-Schwartz A, Del Maestro RF. The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine. PLoS One. 2020 Feb 27;15(2):e0229596. doi: 10.1371/journal.pone.0229596. eCollection 2020.

  PMID: 32106247 BACKGROUND

• Winkler-Schwartz A, Marwa I, Bajunaid K, Mullah M, Alotaibi FE, Bugdadi A, Sawaya R, Sabbagh AJ, Del Maestro R. A Comparison of Visual Rating Scales and Simulated Virtual Reality Metrics in Neurosurgical Training: A Generalizability Theory Study. World Neurosurg. 2019 Jul;127:e230-e235. doi: 10.1016/j.wneu.2019.03.059. Epub 2019 Mar 15.

  PMID: 30880209 BACKGROUND

• Duffy, M.C., et al., Emotions in medical education: Examining the validity of the Medical Emotion Scale (MES) across authentic medical learning environments. Learning and Instruction, 2020. 70: p. 101150.

  BACKGROUND

• Leppink J, Paas F, Van der Vleuten CP, Van Gog T, Van Merrienboer JJ. Development of an instrument for measuring different types of cognitive load. Behav Res Methods. 2013 Dec;45(4):1058-72. doi: 10.3758/s13428-013-0334-1.

  PMID: 23572251 BACKGROUND

• Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.

  PMID: 35191972 DERIVED

Study Officials

• Rolando Del Maestro, MD

  McGill

  PRINCIPAL INVESTIGATOR

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: DOUBLE
• Who Masked: PARTICIPANT, OUTCOMES ASSESSOR
• Masking Details: Participants will not know the purpose of trial
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: SPONSOR

Model Details: Randomized Controlled Trial

Study Record Dates

• First Submitted: January 6, 2021
• First Posted: January 7, 2021
• Study Start: January 15, 2021
• Primary Completion: May 15, 2021
• Study Completion: May 15, 2021
• Last Updated: May 24, 2021

Record last verified: 2021-05

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ICF, CSR, ANALYTIC CODE
• Time Frame: Data will be available for 5 years after completion of trial
• Access Criteria: Researchers wanting access to the data will need to contact the principal investigator of the trial. Dr. Rolando Del Maestro

Locations

CA (1)