Nicolò Buffi, Marco Paciotti, Anthony Gallagher, Pietro Diana, Ruben De Groote, Giovanni Lughezzani, Andrea Gallioli, Paolo Casale, Joan Palou, Alexandre Mottrie, Alberto Breda
 

The general surgical exposure of European urology residents in the performance of invasive procedures, their confidence in the execution, and the overall satisfaction with their training is low. The EuropeaN Training in uRologY (ENTRY) project is a cooperative partnership between Humanitas University, Orsi Academy and Fundació Puigvert, aiming to improve the training of urology residents in minimally invasive procedures. The ENTRY project is developing an educational programme that has three key steps: an eLearning component; a hands-on simulation phase; and a quality-assured methodology for training in the operating room. The final ambition of the project is to provide European urology trainers with high-quality teaching tools and compendia, laying the foundation for a standardized European training programme for residents based on a scientific training approach. The project is supported by the European Commission with an ERASMUS grant.

Stefano Puliatti, Marco Amato, Elio Mazzone, Giuseppe Rosiello, Ruben De Groote, Camille Berquin, Pietro Piazza, Rui Farinha, Alexandre Mottrie, Anthony Gallagher
 

Introduction: The introduction of robot-assisted surgical devices requires the application of objective performance metrics to verify performance levels.
Objective: To develop and validate the performance metrics for a robotic dissection task using a chicken model.
Methods: In a procedure characterization, we developed the performance metrics for a robotic dissection task, using a chicken model. In a modified Delphi panel, 14 experts from four European Union countries agrees on the steps, errors, and critical errors (CEs) of the task. Six experienced surgeons and eight novice urology surgeons performed the robotic dissection task twice on the chicken model. In the Delphi meeting, 100% consensus was reached on five procedure steps, 15 errors and two CEs. Novice surgeons took 20 min to complete the task on trial 1 and 14 min during trial two, whereas experts took 8.2 min and 6.5 min. On average, the Expert Group completed the task 56% faster than the Novice Group and made 46% fewer performance errors.
Results: Sensitivity and specificity for procedure errors and time were excellent to good but poor for step metrics. The mean interrater reliability for the assessments by two robotic surgeons was 0.91 (Expert Group inter-rater reliability = 0.92 and Novice Group = 0.9).
Conclusions: We report evidence which supports the demonstration of face, content and construct validity for a standard and replicable basic robotic dissection task on the chicken model.

Elio Mazzone, Stefano Puliatti, Marco Amato, Brendan Bunting, Bernardo Rocco, Francesco Montorsi, Alexandre Mottrie and Anthony G Gallagher

 

Objective: To analyze all published prospective, randomized and blinded clinical studies on the PBP training using objective performance metrics.
Background: The benefit of 'proficiency-based progression' (PBP) methodology to learning clinical skills in comparison to conventional training is not settled.
Methods: Search of PubMed, Cochrane library‟s Central, EMBASE, MEDLINE and Scopus databases, from inception to 1st March 2020. Two independent reviewers extracted the data. The Medical Education Research Study Quality Instrument (MERSQI) was used to assess the methodological quality of included studies. Results were pooled using biased corrected standardized mean difference and ratio-of-means (ROM). Summary effects were evaluated using a series of fixed and random effects models. The primary outcome was the number of procedural errors performed comparing PBP and non-PBP-based training pathways. Secondary outcomes were the number of procedural steps completed and the time to complete the task/procedure.
Results: From the initial pool of 468 studies, 12 randomized clinical studies with a total of
239 participants were included in the analysis. In comparison to the non-PBP training, ROM results showed that PBP training reduced the number of performance errors by 60% (p < 0.001) and procedural time by 15% (p = 0.003) and increased the number of steps performed by 47% (p < 0.001).
Conclusions and Relevance: Our systematic review and meta-analysis confirms that PBP
training in comparison to conventional or quality assured training improved trainees‟
performances, by decreasing procedural errors and procedural time, while increasing the number of correct steps taken when compared to standard simulation-based training.

Alexandre Mottrie, Elio Mazzone, Peter Wiklund, Markus Graefen, Justin W. Collins,
Ruben De Groote, Paolo Dell’Oglio, Stefano Puliatti, Anthony G Gallagher

 

Background: Identifying objective performance metrics for surgical training in robotic
surgery is imperative for patient safety. Based on this, we aimed to develop and seek
consensus from procedure experts on the metrics which best characterize a reference robotassisted radical prostatectomy (RARP). To determine if the metrics distinguished between the objectively assessed RARP performance of experienced and novice urologists.

Materials and methods: In Study 1, the metrics, i.e., 12 phases of the procedure, 81 steps, 245 errors and 110 critical errors for a reference RARP were developed and then presented to an international Delphi panel of 19 experienced urologists. In Study 2, 12 very experience surgeons (VES) who performed > 500 RARP and 12 novice urology surgeons performed a RARP which was video recorded and assessed by two experienced urologists blinded as to subject and group. Percentage agreement between experienced urologists for the Delphi meeting and Mann-Whitney U and Kruskal-Wallis tests were used for construct validation of the newly identified RARP metrics.

Results: At the Delphi panel, consensus was reached on the appropriateness of the metrics for a reference RARP. In Study 2 the results showed that the VES performed ~4% more procedure steps and made 72% fewer procedure errors than the novice group (p = 0.027). Phases 7a & b (i.e., neurovascular bundle dissection) best discriminated between the VES and Novice surgeons. Limitations: VES whose performance was in the bottom half of their group demonstrated considerable error variability, made five-times as many errors as the other half of the group (p = 0.006).

​

Conclusions: The international Delphi panel reached high-level consensus on the RARP
metrics which reliably distinguished between the objectively scored procedure performance of VES and novice RARP surgeons. Reliable and valid performance metrics of robotic prostatectomy are imperative for effective and quality assured surgical training.

Stefano Puliatti, Elio Mazzone, Marco Amato, Ruben De Groote, Alexandre Mottrie & Anthony G. Gallagher

 

Background To improve patient safety, there is an imperative to develop objective performance metrics for basic surgical skills training in robotic surgery.

Objective To develop and validate (face, content, and construct) the performance metrics for robotic suturing and knot tying, using a chicken anastomotic model.

Design, setting and participants Study 1: In a procedure characterization, we developed the performance metrics (i.e., procedure steps, errors, and critical errors) for robotic suturing and knot tying, using a chicken anastomotic model. In a modified Delphi panel of 13 experts from four EU countries, we achieved 100% consensus on the five steps, 18 errors and four critical errors (CE) of the task.

Study 2: Ten experienced surgeons and nine novice urology surgeons performed the robotic suturing and knot tying chicken anastomotic task. The mean inter-rater reliability for the assessments by two experienced robotic surgeons was 0.92 (95% CI, 0.9–0.95). Novices took 18.5 min to complete the task and experts took 8.2 min. (p = 0.00001) and made 74% more objectively assessed performance errors than the experts (p = 0.000343).

Conclusions We demonstrated face, content, and construct validity for a standard and replicable basic anastomotic robotic suturing and knot tying task on a chicken model.

Patient summary Validated, objective, and transparent performance metrics of a robotic surgical suturing and knot tying tasks are imperative for effective and quality assured surgical training.

Aude E. Vanlander, Elio Mazzone, Justin W. Collins, Alexandre M. Mottrie,
Xavier M. Rogiers, Henk G. van der Poel, Isabelle Van Herzeele, Richard M. Satava,
Anthony G. Gallagher

 

To improve patient outcomes in robotic surgery, robotic training and education need to be
modernised and augmented. The skills and performance levels of trainees need to be
objectively assessed before they operate on real patients. The main goal of the
first Orsi Consensus Meeting on European Robotic Training (OCERT) was to establish the opinions of experts from different scientific societies on standardised robotic training pathways and training methodology. After a 2-d consensus conference, 36 experts identified 23 key statements allotted to three themes: training standardisation pathways, validation metrics, and implementation prerequisites and certification. After two rounds of Delphi voting, consensus was obtained for 22 of 23 questions among these three categories. Participants agreed that societies should drive and support the implementation of benchmarked training using validated proficiency-based pathways. All courses should deliver an internationally agreed curriculum with performance standards, be accredited by universities/professional societies, and, trainees should receive a certificate approved by professional societies and/or universities after successful completion of the robotic training courses. This OCERT meeting established a basis for bringing surgical robotic training out of the operating room by seeking input and consensus across surgical specialties for an objective, validated, and standardised training programme with transparent, metric-based training outcomes.