Randomised controlled trial to assess the effect of a Just-in-Time training on procedural performance: A proof-of-concept study to address procedural skill decay


Background: A subset of high-risk procedures present significant safety threats due to their (1) infrequent occurrence, (2) execution under time constraints and (3) immediate necessity for patient survival. A Just-in-Time (JIT) intervention could provide real-time bedside guidance to improve high-risk procedural performance and address procedural deficits associated with skill decay.

Objective: To evaluate the impact of a novel JIT intervention on transvenous pacemaker (TVP) placement during a simulated patient event.

Methods: This was a prospective, randomised controlled study to determine the effect of a JIT intervention on performance of TVP placement. Subjects included board-certified emergency medicine physicians from two hospitals. The JIT intervention consisted of a portable, bedside computer-based procedural adjunct. The primary outcome was performance during a simulated patient encounter requiring TVP placement, as assessed by trained raters using a technical skills checklist. Secondary outcomes included global performance ratings, time to TVP placement, number of critical omissions and System Usability Scale scores (intervention only).

Results: Groups were similar at baseline across all outcomes. Compared with the control group, the intervention group demonstrated statistically significant improvement in the technical checklist score (11.45 vs 23.44, p < 0.001, Cohen’s d effect size 4.64), the global rating scale (2.27 vs 4.54, p < 0.001, Cohen’s d effect size 3.76), and a statistically significant reduction in critical omissions (2.23 vs 0.68, p < 0.001, Cohen’s d effect size -1.86). The difference in time to procedural completion was not statistically significant between conditions (11.15 min vs 12.80 min, p = 0.12, Cohen’s d effect size 0.65). System Usability Scale scores demonstrated excellent usability.

Conclusion: A JIT intervention improved procedure perfromance, suggesting a role for JIT interventions in rarely performed procedures.

BMJ Quality & Safety, 26, 881-891
Elizabeth D. Rosenman
Elizabeth D. Rosenman
Assistant Professor, Emergency Medicine & Director of Simulation
James A. Grand
James A. Grand
Associate Professor, Psychology

I’m a scientist at heart, an organizational psychologist by training, and a lucky dad and husband all the time.

Rosemarie Fernandez
Rosemarie Fernandez
Clincal Associate Professor, Emergency Medicine