Effect of Intelligent Tutor Induced Pausing on Learning Simulated Surgical Skills

NCT06235788 | Recruiting

• 1 other identifier: 2010-270, NEU-09-042-Trial 4
• Study Type: interventional
• Target: 129
• Locations: 1 country
• Sites: 1

Brief Summary

Traditional training of surgical technical skills relies on mentorship from experienced surgeons, who continuously evaluate and change trainee performance to prevent errors and potential patient harm by providing verbal instructions. These educators may also pause the procedure, explaining the risks associated with the trainee's actions, and may personally demonstrate proper techniques to the students. Studies examining pausing while providing medical care outline that these approaches allow for learning. An artificial intelligent (AI) tutoring system, the Intelligent Continuous Expertise Monitoring System (ICEMS), improves learning in a surgical simulated operation by providing trainees with verbal instructions upon error identification. However, the effect of including a pause during this AI teaching has not been studied. Therefore, the ICEMS post-error identification methodology has been altered to include a pause with the intelligent tutor voice instruction. The aim of this study is to determine the effect of pausing on surgical skill acquisition and transfer among pre-medical and medical students. This will be done by comparing their performance in repeated simulated tumour resection tasks.

Conditions

Surgical Education

Keywords

Artificial intelligence; Intelligent tutor; Clinical trial; Medical students; Competency-based feedback; Neurosurgery; Virtual reality

Outcome Measures

Primary Outcomes (3)

• Change in performance

  Evaluated by comparing the average composite-score, calculated by the ICEMS, from each practice scenario. Scores range from expert/skilled level (a score of 1.00) to novice/less-skilled level (a score of -1.00).

  1 day of study

• Transfer of learning

  Evaluated by comparing the average composite-score, calculated by the ICEMS, from each practice scenario. Scores range from expert/skilled level (a score of 1.00) to novice/less-skilled level (a score of -1.00).

  1 day of study

• Objective Structured Assessment of Technical Skills (OSATS) global rating scale

  Performance score of the participants in the complex realistic scenario, assessed by two blinded experts using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale on a 7-point Likert scale (1= novice to 7 = expert). Efficacy in learning with pausing methodology and an expert-level demonstration video will be compared to pausing methodology alone and to no pausing methodology.

  1 day of study

Secondary Outcomes (2)

• Differences in strength of emotions elicited

  1 day of study

• Difference in Cognitive Load

  1 day of study

Study Arms (3)

Control Group Intelligent Continuous Expertise Monitoring System (ICEMS) only verbal feedback group

NO INTERVENTION

43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario. Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.

Experimental Group ICEMS verbal feedback with pause group

EXPERIMENTAL

43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario. Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.

Behavioral: Experimental Group ICEMS verbal feedback with pause group

Experimental Group ICEMS verbal feedback with pause and expert-level video demonstration

EXPERIMENTAL

43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario. Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.

Behavioral: Experimental Group ICEMS audio feedback with pause and expert video

Interventions

Experimental Group ICEMS verbal feedback with pause group BEHAVIORAL

While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down and reflecting for 22 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

Experimental Group ICEMS verbal feedback with pause group

Experimental Group ICEMS audio feedback with pause and expert video BEHAVIORAL

While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down, turning their attention to a 9-second expert-level demonstration video, and reflecting for 13 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

Experimental Group ICEMS verbal feedback with pause and expert-level video demonstration

Eligibility Criteria

• Age: 18 Years+
• Sex: all
• Healthy Volunteers: Yes
• Age Groups: Adult (18-64), Older Adult (65+)

You may not qualify if:

• Participation in previous trials involving the NeuroVR (CAE Healthcare) simulator

Sponsors & Collaborators

• McGill University: lead

Study Sites (1)

Neurosurgical Simulation and Artificial Intelligence Learning Centre

Montreal, Quebec, H2X 4B3, Canada

RECRUITING

Related Publications (4)

• Yilmaz R, Winkler-Schwartz A, Mirchi N, Reich A, Christie S, Tran DH, Ledwos N, Fazlollahi AM, Santaguida C, Sabbagh AJ, Bajunaid K, Del Maestro R. Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation. NPJ Digit Med. 2022 Apr 26;5(1):54. doi: 10.1038/s41746-022-00596-8.

  PMID: 35473961 BACKGROUND

• Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.

  PMID: 35191972 BACKGROUND

• Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.

  PMID: 31373651 BACKGROUND

• Lee JY, Szulewski A, Young JQ, Donkers J, Jarodzka H, van Merrienboer JJG. The medical pause: Importance, processes and training. Med Educ. 2021 Oct;55(10):1152-1160. doi: 10.1111/medu.14529. Epub 2021 May 1.

  PMID: 33772840 BACKGROUND

Study Officials

• Rolando F Del Maestro, MD, PhD

  McGill University

  PRINCIPAL INVESTIGATOR

Central Study Contacts

Rolando F Del Maestro, MD, PhD

CONTACT

519 708 0346; rolando.del_maestro@mcgill.ca

Vanja Davidovic

CONTACT

6132964855; vanja.davidovic@mail.mcgill.ca

Study Design

• Study Type: interventional
• Phase: not applicable
• Allocation: RANDOMIZED
• Masking: DOUBLE
• Who Masked: PARTICIPANT, OUTCOMES ASSESSOR
• Masking Details: Double (Participant and Expert Rater) Participants are unaware of the intervention being used in the study - the pause. They are informed that they will be acquiring and honing technical skills relevant to neurosurgery, with the added feature of receiving feedback from an intelligent system during subpial tumour resection simulations. Experts, when providing OSATS ratings, do not know to which group the video performance they are rating belongs.
• Purpose: HEALTH SERVICES RESEARCH
• Intervention Model: PARALLEL
• Sponsor Type: OTHER
• Responsible Party: PRINCIPAL INVESTIGATOR
• PI Title: Director, Neurosurgical Simulation and Artificial Intelligence Learning Centre

Model Details: Randomized Controlled Trial

Study Record Dates

• First Submitted: January 23, 2024
• First Posted: February 1, 2024
• Study Start: January 30, 2024
• Primary Completion: September 1, 2024
• Study Completion: September 1, 2024
• Last Updated: February 5, 2024

Record last verified: 2024-02

Data Sharing

• IPD Sharing: Will share
• Shared Documents: STUDY PROTOCOL, SAP, ICF, CSR, ANALYTIC CODE
• Time Frame: Data will be available for 5 years after completion of trial.
• Access Criteria: Researchers wanting access to the data will need to contact the principal investigator of the trial. Dr. Rolando Del Maestro

Locations

CA (1)