Virtual reality training in coronary angiography and its transfer effect to real-life catheterisation lab

Abstract

Aims: The aim of this study was to explore if proficiency-based training in a coronary angiography (CA) simulator can transfer acquired skills from virtual reality (VR) to the real world in order to improve early performance.

Methods and results: Sixteen senior cardiology residents were randomised to proficiency-based VR training or control. Two consecutive CAs were performed on patients. Skills metrics and errors were compared between the groups. Thirty-two CAs were performed under the supervision of an experienced interventionalist. VR-trained residents practised for a mean of 10 hours in a CA simulator. In real life, the VR-trained group had shorter fluoroscopy and total procedure times than the controls (median 558 vs. 842 seconds, p=0.003 and 1,356 vs. 1,623 seconds, p=0.032, respectively). The controls had a higher error score (median 27 vs. 15, p=0.002) and a lower performance score (median 47 vs. 68, p=0.006) than the VR-trained residents.

Conclusions: Simulator-based training in CA improved skills and decreased errors compared to mentor-based training only. CA training in VR resulted in a superior performance, measured by fluoroscopy and total procedure times, and superior error and performance scores, thereby confirming transfer validity. Our recommendation is to incorporate VR training in the curriculum for the general cardiologist to improve safe learning in CA.

Abbreviations

ACC: American College of Cardiology

CA: coronary angiography

CABG: coronary artery bypass grafting

DAP: dose area product

ESC: European Society of Cardiology

IQR: interquartile range

LCA: left coronary artery

MRT: mental rotation test

OR: operation room

RCA: right coronary artery

SCAAR: Swedish Coronary Angiography and Angioplasty Registry

VIST®: vascular intervention simulation trainer

VR: virtual reality

Introduction

Mentor-based training on patients is the gold standard for skills acquisition in coronary angiography (CA). Stepwise progression from a standby operator to a solo CA interventionalist takes a long time and is associated with an increased number of complications for the patient and radiation exposure for the patient and the cathlab staff1-6. A safe way of training residents in cardiology in cardiac interventions is warranted and simulators are proposed as being of value in skills acquisition in the present European Society of Cardiology (ESC) guidelines7. For the general cardiologist, between 100 and 300 CAs are recommended to be performed during training according to American College of Cardiology (ACC) and ESC guidelines7,8. The recommendation for methods of training is, however, vague, and assessment of acquired skills is often absent. Training in a safe environment, both for the patient and for the trainee, up to a predefined expert performance level to accelerate the learning curve on the patient, would be of great importance.

Despite the presence of simulators for training in coronary interventions for a decade, none has demonstrated transferability from virtual reality (VR) to the real world in a randomised setting in beginners. Validating simulators includes not only demonstrating that the simulator can discriminate between different performance levels (construct validity) but also that the skills acquired in VR can be transferred to the real world with an accelerated learning curve with the aim of a reduced number of complications (transfer validity). So far, successful attempts at transfer validation have been performed, mainly in peripheral vascular interventions and in endoscopic surgery and, without doubt, VR training works with some of the current simulators9-12. Recently, the Mentice VIST® simulator (Mentice, Gothenburg, Sweden) improved technical performance after limited non-proficiency-based CA training in the simulator in a group of cardiology fellows with experience in CA ranging from beginners to experts13. Not all simulators automatically improve behaviour in the real world, a fact that is supported by several studies and meta-analyses14-19. We have made a series of studies to solve the question as to whether simulator training in CA is beneficial. We started out to define factors important for proficiency in real world CA and concluded that fluoroscopy time was the only metric following a learning curve in acquiring skills in CA1. This was followed by a construct validation attempt of the Mentice VIST CA simulator, showing that it can discriminate among beginners, intermediates and experts20. A randomised study exploring proficiency-based VR training in true beginners and transferability to the real world is the missing link to recommend or discard simulator training in CA definitively. Our hypothesis was that proficiency-guided simulator-based training would accelerate learning when starting real-world CA.

Methods

During 2011-2012, fifty-four cardiology residents from the Stockholm metropolitan area were invited to take part in this randomised proficiency-based simulator training trial. Thirteen had previous experience in CA simulator training and were thus excluded. Twenty-one residents did not respond or were not interested in participation. Twenty senior cardiology residents finally volunteered to participate. After a stratified randomisation, the simulator group practised in a CA simulator up to a predefined proficiency level with at least eight hours of practice. All participants performed two video-filmed supervised CA on patients.

STUDY SUBJECTS

All participants were cardiology residents from four different hospitals in the Stockholm metropolitan area in their second half of a five-year cardiology training programme. The baseline characteristics of the participants are shown in Table 1. None had any experience in performing CA or experience in CA simulation training. Four participants were not able to complete the study (Figure 1).

Figure 1. Study flow chart. CA: coronary angiography; MRT: mental rotation test

WEB-BASED COURSE

A web-based course on CA was created to standardise the cognitive learning for the study participants. The course was available online (www.coronaryintervention.org) and featured chapters for anatomy, pharmacology, complications, puncture technique, radiation safety and materials. A chapter also explained how to perform the full CA procedure on a patient, from arterial puncture to arterial closure.

PRE-TEST ASSESSMENT

A written exam, including 25 multiple-choice questions, based on the content of the web-based course, was taken by all participants at a start-up meeting. A mental rotation test (MRT) was carried out to test the participants’ visual-spatial abilities. A survey was completed to reveal previous simulator and catheterisation experience, and experience in video gaming.

RANDOMISATION PROCEDURE

Based on age and gender, the results from the written exam, the MRT test and the survey regarding previous experience with cardiac catheterisation, simulation or video games, the participants were matched in pairs and thereafter randomised to simulator training or control by the sealed envelope method.

SIMULATOR

The Mentice VIST system is a vascular intervention trainer including a CA module. This simulator is a full-scale simulator enabling the entire procedure to be practised, except introducer insertion and flushing of catheters. The interface includes a mannequin, two monitors, and joysticks for table and C-arm control. The simulator is provided with buttons for zooming and X-ray intensity, and pedals for fluoroscopy and cine loop control (Figure 2). The machine accepts real interventional tools such as wires and catheters after the tip is cut off. The properties of the interventional tools, X-ray and cine loops are all simulated. Virtual contrast is created by injecting air by a syringe. Virtual haptics are produced by the simulator to get the sensation of tactile feedback. In this version of the software, a total of 31 anatomical coronary cases with different properties regarding the anatomy of the aortic root and coronary lesions are available to be chosen from.

Figure 2. Mentice VIST® simulator used in the study.

SIMULATOR TRAINING

The simulator group received demonstration and hands-on training on the Mentice VIST on how to perform the CA procedure, including safe handling of catheters. Simulator training was proficiency-guided based on a study defining proficiency level in the simulator when performed by 10 expert CA interventionalists20. The proficiency level definition was: performing a complete CA with eight standardised cine loops in less than ten minutes, fluoroscopy time less than three minutes and using less than 50 ml of contrast. The minimum training requirement was eight hours of unsupervised VR training with repeated tests of proficiency level.

CORONARY ANGIOGRAPHY (CA)

The four-step model for teaching skills of “see how”, “show how”, “tell how” and “does” was practised by all participants. Prior to performing CA on patients, all participants were instructed to participate in at least two CAs performed by experienced operators at their home hospital to understand how the procedure is performed and how to act and dress in the cathlab (“see how”). Before the participants were introduced to patients, all were instructed on how to perform the procedure and what to pay attention to regarding safety in catheter, radiation and contrast handling. Technical issues regarding table and C-arm handling were explained and demonstrated to the trainees (“show how”). Participants also had to explain how to perform a complete CA (“tell how”).

Two consecutive CAs were performed by each participant while video filming the monitors with voice recording to reveal oral help from the supervisor (“does”). The supervisor (Per Tornvall) was blinded to randomisation and was instructed to provide oral advice to the trainee if there was lack of progress or dangerous behaviour risking patient safety. Manual help was only provided if oral advice did not result in sufficient improvement or progress. All 32 CAs were supervised by the same expert. Only the procedural steps trainable in the simulator were performed by the participants. Excluded parts were arterial puncture and flushing of the catheters since these steps were not included in the virtual practice in Mentice VIST. The femoral access route was used in all patients to mimic the default route in the simulator. Introduction of catheters and advancing them from the introducer to the coronary ostia, handling of the C-arm and table were performed solely by the participant. Contrast delivery to the patient was also handled by the trainee and administered through an injection pump (ACIST CVi® Contrast Delivery System; ACIST® Medical Systems Inc., Eden Prairie, MN, USA). Total procedure time, fluoroscopy time, radiation dose (dose area product [DAP]), number of cine loops and the amount of contrast was recorded, as well as the amount and type of advice and help provided by the supervisor. All procedures were performed without a radiography technician on a single plane cardiovascular X-ray system (Allura Xper FD10; Philips Healthcare, Best, The Netherlands). All participants acquired a radiation safety certificate on completion of the web course.

STUDY PATIENTS

CA was performed on patients scheduled for planned or subacute CA after informed consent. Patients eligible for the study were investigated for coronary heart disease and diagnosed with angina or valvular heart disease (Table 1). Previous CABG patients were included but the assessed procedure only consisted of CA of the native vessels.

EVALUATION OF CA PERFORMANCE

Video recordings of the monitors were obtained during the CA procedures with a digital HD video camera (Sony® HDR-TG3E; Sony, Tokyo, Japan). All recordings were analysed by an invasive cardiologist with experience of performing >4,000 CA procedures (Jens Jensen). The procedures were evaluated regarding procedure quality and safety by two different scores (Table 2, Table 3). Assessment of error scoring has been described elsewhere1,9,20,21. Performance and ability scoring has been reported in cardiac transseptal puncture and a modification of this score was made to fit the CA procedure12.

STATISTICAL ANALYSIS

Data are presented as median and interquartile range (IQR), mean or median (range) and numbers (%). Descriptive summary statistics were used when appropriate. Differences between the groups were tested with the Mann-Whitney U test or chi-square test. Analyses were performed using Statistica version 10 (Statsoft Inc., Tulsa, OK, USA). We aimed to include between 15 and 20 residents in cardiology in the study. With 16 participants we calculated to have 80% statistical power (p<0.05) to detect a difference in fluoroscopy time between the simulator-trained and control groups of at least three minutes, or to have at least a 25% lower error score.

ETHICS

All participants received written information about the project declaring anonymity in the video recordings and evaluation. Only the monitors were filmed and sound was recorded. The protocols and procedures were approved by the local ethical committee for human research at Karolinska Institutet, Stockholm, Sweden (ref. nr. 04-202/1). The study was performed according to the Declaration of Helsinki regarding good clinical practice. Informed consent was acquired from all participating residents and patients.

Results

PRE-TEST ASSESSMENT

There were no age or gender differences between the groups. Mean test score on the web-based course was 25/31 (range 22-29) correct answers. MRT test scores were equal and experience of catheterisation or medical simulators did not differ between the groups (Table 1).

SIMULATOR TRAINING

Eight trainees practised for a mean 10 hours and 12 min (range eight hours and 23 min-13 hours and 36 min) in 31 (range 28-31) different coronary anatomies over seven (range three-10) training sessions. All simulator trainees reached expert level and were thus able to reproduce expert performance under supervision before performing CA on patients.

CA ASSESSMENT

Thirty-two video-filmed CAs were performed on 32 patients. VR-trained residents had shorter fluoroscopy and total procedure times (Figure 3A, Figure 3B) and fewer cine loops (Figure 3C) than controls. No differences were seen in contrast volume or radiation dose (Table 4). Patient case mix was comparable between the groups (Table 1). The VR-trained group outperformed the control group regarding procedure safety and quality (Figure 4A, Figure 4B).

Figure 3. Results. A) Fluoroscopy time. B) Total procedure time. C) Number of cine loops recorded. Box-and-whisker plots of metrics recorded at the 32 CA performed. Values in median (IQR). Differences tested by Mann-Whitney U test.

Figure 4. Results. A) Error score. B) Total performance score. Box-and-whisker plots of error and performance score of the assessment of 32 CA video recordings by a blinded assessor. Values in median (IQR). Differences tested by Mann-Whitney U test.

Estimated probability to perform a CA within suggested fluoroscopy and total procedure times with good quality in performance and a low error score was highest in simulator trainees with good visual-spatial ability (Table 5). No complications associated with the procedure appeared in the cathlab or on the ward.

Discussion

For the first time transfer validation was demonstrated in VR-trained beginners in CA with the strongest association to the metric fluoroscopy time. Fluoroscopy time has previously been demonstrated to have a well-defined learning curve, with beginners reaching expert level after about 150 CAs and being associated with complications1. Simulator-trained residents also performed CA with fewer errors and need for supervisor’s advice or help with better quality measured by a performance score. No complications occurred in the cathlab or on the ward but data from SCAAR indicated that prolonged fluoroscopy time increased the likelihood for complications, and an extended series of CA might have revealed an increased complication risk in the conventionally trained group1. The impact of preparatory simulator training on the number of medical errors worldwide could be great. An estimation is that at least two million CAs are performed annually in the world. The complication rate is higher in inexperienced operators reaching at least 1.3%3. One can speculate that reducing this rate by 25% to 1% complications by simulator training would result in a reduction in the number of complications worldwide by 6,000, a fact that makes it an important health and cost issue.

PRE-TEST ASSESSMENT AND RANDOMISATION

The randomisation procedure aimed for similar groups to adjust for potential known confounders. MRT was performed to test the residents’ visual-spatial abilities, previously described as affecting simulator performance22. Visual-spatial abilities, known to be gender-specific, were adjusted for in the matching procedure23. Age and previous experience in simulators or cathlab could also affect outcome, and the matching aimed for compensation of these factors. There were no pre-test differences between the groups and also no differences between the genders regarding visual-spatial ability, and this is a good reason to believe that the groups were comparable.

SIMULATOR TRAINING

An expert level in VR CA performance was reached after a fairly short training time in the simulator. Proficiency in the simulator was settled after 10 expert interventionalists had performed five VR CA each20. The minimum practising requirement for reaching simulator proficiency was estimated to be eight hours of VR training, and the residents fulfilled this requirement by practising in median 10 hours. Training sessions in the simulator were unsupervised but residents were able to reproduce and demonstrate proficiency in the simulator prior to performing CA on patients, hence assuring acquired skills and safe behaviour. Bagai et al randomised fellows in cardiology with a variety of experience in CA to simulator training or no training13. More than 50% of these participants had performed >50 CA and were thus not true beginners. The simulator training was not proficiency-guided and lasted for only two to four hours. Despite this, the simulator-trained fellows gained in technical performance. In contrast, our study involved a gold standard method of expert proficiency-based simulator training and improved the simulator trainees’ skills associated with complications as well as quality measures.

TRANSFER ASSESSMENT

Assessing practical skills is always associated with some degree of subjectivity. Therefore, we aimed for a method which was as objective as possible. Numerical metrics from the CA procedure such as time, contrast, number of cine loops and radiation are objective, and the VR-trained group demonstrated superiority in all but radiation and contrast. However, contrast is probably not associated with proficiency, since a registry study failed to reveal a learning curve or distinct proficiency level in this metric1. The quality of the procedure was also measured by catheter handling, quality and visualisation of the coronary anatomy. A somewhat more subjective scale used by others12 was used for this assessment. The scaling, to be as objective as possible, was divided into key steps for a more precise evaluation (Table 2, Table 3). Simulator trainees outperformed the control group in these more subjective metrics as well, which illustrates safe behaviour.

Despite relatively short training time in the simulator, there was a clear benefit when added to mentor-based learning. The reason for this superiority can be explained by familiarity with the technique obtained during simulator training causing automation of key procedural steps when performing the procedure in real life. This might result in increased mental concentration on manoeuvring catheters and producing accurate cine loops, leading to safer and better investigations. Participants most likely to perform well were simulator-trained participants with high MRT scores and, for future recruitment of interventionalists, one might take this into consideration. Supervised VR training, though time-consuming and costly, might enhance the learning process, and should be tested in future studies.

STUDY LIMITATIONS

The number of trainees in this study was relatively small, but the benefit from VR training in CA was not known and therefore made power calculation difficult. We estimated that an improvement of at least 25% in fluoroscopy time or 25% fewer errors in the simulator group would be of clinical significance. All residents eligible and willing to participate in the Stockholm metropolitan area were included and to conduct a larger study would have meant a large multicentre study. However, there is only one other similar simulator in Sweden which is unfortunately not in clinical use and without support.

Also, the number of procedures performed on patients was small due to logistic reasons. It was therefore impossible to show learning curves for the two groups. However, learning curves will show in any procedure, and the issue was whether simulator training improved the initial behaviour in the cathlab. To detect differences in learning curves between the two groups one would have to analyse an extended series of CA, maybe up to 150 cases based on the experience of previous studies on learning curves in CA1.

There was only one expert assessing the video recordings and therefore it was not possible to calculate inter-rater reliability. However, in the construct validity study of the present simulator we had two experts rating the performances with high inter-rater reliability20.

Conclusion

In conclusion, preparatory proficiency-based CA training in VR resulted in a superior real-life performance when added to conventional mentor-based training regarding quality and safety. Our recommendation is to incorporate VR training in the curriculum for the general cardiologist to improve safe learning of CA.

Acknowledgements

The staff at KTC (clinical training centre) and CAMST (Center for Advanced Medical Simulation and Training). All cardiology residents in this trial and their hospitals for giving them time off to participate. The cathlab staff for patience and patients for attendance.

Funding

This study was funded by fees from simulator courses conducted previously at the institution. No external funding existed.

Conflict of interest statement

The authors have no conflicts of interest to declare.