Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

R D Dias; M C Ngo-Howard; M T Boskovski; M A Zenati; S J Yule

doi:10.1002/bjs.10795

journal article Feb 21, 2018

Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

R D Dias

M C Ngo-Howard M T Boskovski M A Zenati S J Yule

British Journal of Surgery Vol. 105 No. 5 pp. 491-501 · Oxford University Press (OUP)

View at Publisher Save 10.1002/bjs.10795

Abstract

Abstract

Background
Surgeons in the operating theatre deal constantly with high-demand tasks that require simultaneous processing of a large amount of information. In certain situations, high cognitive load occurs, which may impact negatively on a surgeon's performance. This systematic review aims to provide a comprehensive understanding of the different methods used to assess surgeons' cognitive load, and a critique of the reliability and validity of current assessment metrics.

Methods
A search strategy encompassing MEDLINE, Embase, Web of Science, PsycINFO, ACM Digital Library, IEEE Xplore, PROSPERO and the Cochrane database was developed to identify peer-reviewed articles published from inception to November 2016. Quality was assessed by using the Medical Education Research Study Quality Instrument (MERSQI). A summary table was created to describe study design, setting, specialty, participants, cognitive load measures and MERSQI score.

Results
Of 391 articles retrieved, 84 met the inclusion criteria, totalling 2053 unique participants. Most studies were carried out in a simulated setting (59 studies, 70 per cent). Sixty studies (71 per cent) used self-reporting methods, of which the NASA Task Load Index (NASA-TLX) was the most commonly applied tool (44 studies, 52 per cent). Heart rate variability analysis was the most used real-time method (11 studies, 13 per cent).

Conclusion
Self-report instruments are valuable when the aim is to assess the overall cognitive load in different surgical procedures and assess learning curves within competence-based surgical education. When the aim is to assess cognitive load related to specific operative stages, real-time tools should be used, as they allow capture of cognitive load fluctuation. A combination of both subjective and objective methods might provide optimal measurement of surgeons' cognition.

Topics

No keywords indexed for this article. Browse by subject →

References

118

[1]

Tavare "Where are we with transparency over performance of doctors and institutions?" BMJ (2012) 10.1136/bmj.e4464

[2]

Madani "What are the principles that guide behaviors in the operating room?: creating a framework to define and measure performance" Ann Surg (2017) 10.1097/sla.0000000000001962

[3]

Landon "Physician clinical performance assessment: prospects and barriers" JAMA (2003) 10.1001/jama.290.9.1183

[4]

Rogers "Teaching operating room conflict management to surgeons: clarifying the optimal approach" Med Educ (2011) 10.1111/j.1365-2923.2011.04040.x

[5]

Hyder "Performance measurement in surgery through the National Quality Forum" J Am Coll Surg (2014) 10.1016/j.jamcollsurg.2014.06.018

[6]

Christian "A prospective study of patient safety in the operating room" Surgery (2006) 10.1016/j.surg.2005.07.037

[7]

Modi "Temporal stress in the operating room: brain engagement promotes ‘Coping’ and disengagement prompts ‘Choking’" Ann Surg (2017)

[8]

Maruthappu "Systematic review of methodological quality of individual performance measurement in surgery" Br J Surg (2014) 10.1002/bjs.9642

[9]

Young "Cognitive load theory: implications for medical education: AMEE guide no. 86" Med Teach (2014) 10.3109/0142159x.2014.889290

[10]

Van Merriënboer "Cognitive load theory in health professional education: design principles and strategies" Med Educ (2010) 10.1111/j.1365-2923.2009.03498.x

[11]

The magical number seven, plus or minus two: Some limits on our capacity for processing information.

George A. Miller

Psychological Review 1956 10.1037/h0043158

[12]

Sweller "Cognitive load during problem solving: effects on learning" Cogn Sci (1988) 10.1207/s15516709cog1202_4

[13]

Yurko "Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool" Simul Healthc (2010) 10.1097/sih.0b013e3181e3f329

[14]

Schuetz "Three different types of surgeon-specific stress reactions identified by laparoscopic simulation in a virtual scenario" Surg Endosc (2008) 10.1007/s00464-007-9605-1

[15]

Haji "Measuring cognitive load: performance, mental effort and simulation task complexity" Med Educ (2015) 10.1111/medu.12773

[16]

Eversbusch "Learning curves and impact of psychomotor training on performance in simulated colonoscopy: a randomized trial using a virtual reality endoscopy trainer" Surg Endosc (2004) 10.1007/s00464-003-9264-9

[17]

Carswell "Assessing mental workload during laparoscopic surgery" Surg Innov (2005) 10.1177/155335060501200112

[18]

Naismith "Validity of cognitive load measures in simulation-based training: a systematic review" Acad Med (2015) 10.1097/acm.0000000000000893

[19]

Moher "Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement" BMJ (2009) 10.1136/bmj.b2535

[20]

Shamseer "Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation" BMJ (2015) 10.1136/bmj.g7647

[21]

Harris "Research Electronic Data Capture (REDCap) – a metadata-driven methodology and workflow process for providing translational research informatics support" J Biomed Inform (2009) 10.1016/j.jbi.2008.08.010

[22]

Reed "Association between funding and quality of published medical education research" JAMA (2007) 10.1001/jama.298.9.1002

[23]

Richstone "Eye metrics as an objective assessment of surgical skill" Ann Surg (2010) 10.1097/sla.0b013e3181e464fb

[24]

Nasa-Task Load Index (NASA-TLX); 20 Years Later

Sandra G. Hart

Proceedings of the Human Factors and Ergonomics So... 2006 10.1177/154193120605000909

[25]

Wilson "Development and validation of a surgical workload measure: the surgery task load index (SURG-TLX)" World J Surg (2011) 10.1007/s00268-011-1141-4

[26]

Weigl "The impact of intra-operative interruptions on surgeons' perceived workload: an observational study in elective general and orthopedic surgery" Surg Endosc (2015) 10.1007/s00464-014-3668-6

[27]

Berg "The impact of heat stress on operative performance and cognitive function during simulated laparoscopic operative tasks" Surgery (2015) 10.1016/j.surg.2014.06.012

[28]

Weigl "Intra-operative disruptions, surgeon's mental workload, and technical performance in a full-scale simulated procedure" Surg Endosc (2016) 10.1007/s00464-015-4239-1

[29]

Yu "Intraoperative workload in robotic surgery assessed by wearable motion tracking sensors and questionnaires" Surg Endosc (2017) 10.1007/s00464-016-5047-y

[30]

Yu "Quantifying intraoperative workloads across the surgical team roles: room for better balance?" World J Surg (2016) 10.1007/s00268-016-3449-6

[31]

Wucherer "Vertebroplasty performance on simulator for 19 surgeons using hierarchical task analysis" IEEE Trans Med Imaging (2015) 10.1109/tmi.2015.2389033

[32]

Zheng "Mastering instruments before operating on a patient: the role of simulation training in tool use skills" Surg Innov (2014) 10.1177/1553350614532533

[33]

Bertolaccini "Ergon-trial: ergonomic evaluation of single-port access versus three-port access video-assisted thoracic surgery" Surg Endosc (2015) 10.1007/s00464-014-4024-6

[34]

Dixon "Augmented real-time navigation with critical structure proximity alerts for endoscopic skull base surgery" Laryngoscope (2014) 10.1002/lary.24385

[35]

Foo "Evaluating mental workload of two-dimensional and three-dimensional visualization for anatomical structure localization" J Laparoendosc Adv Surg Tech A (2013) 10.1089/lap.2012.0150

[36]

Britt "Intracorporeal suturing: transfer from Fundamentals of Laparoscopic Surgery to cadavers results in substantial increase in mental workload" Surgery (2015) 10.1016/j.surg.2015.03.032

[37]

Chowriappa "Augmented-reality-based skills training for robot-assisted urethrovesical anastomosis: a multi-institutional randomised controlled trial" BJU Int (2015) 10.1111/bju.12704

[38]

Gardner "Impact of seductive details on the acquisition and transfer of laparoscopic suturing skills: emotionally interesting or cognitively taxing?" Surgery (2016) 10.1016/j.surg.2016.05.015

[39]

Porges "The polyvagal perspective" Biol Psychol (2007) 10.1016/j.biopsycho.2006.06.009

[40]

Thayer "Claude Bernard and the heart–brain connection: further elaboration of a model of neurovisceral integration" Neurosci Biobehav Rev (2009) 10.1016/j.neubiorev.2008.08.004

[41]

Gianaros "Regional cerebral blood flow correlates with heart period and high-frequency heart period variability during working-memory tasks: implications for the cortical and subcortical regulation of cardiac autonomic activity" Psychophysiology (2004) 10.1111/1469-8986.2004.00179.x

[42]

Thayer "Heart rate variability, prefrontal neural function, and cognitive performance: the neurovisceral integration perspective on self-regulation, adaptation, and health" Ann Behav Med (2009) 10.1007/s12160-009-9101-z

[43]

Rieger "Heart rate and heart rate variability as indirect markers of surgeons' intraoperative stress" Int Arch Occup Environ Health (2014) 10.1007/s00420-013-0847-z

[44]

Rieger "Psychophysical workload in the operating room: primary surgeon versus assistant" Surg Endosc (2015) 10.1007/s00464-014-3899-6

[45]

Böhm "A prospective randomized trial on heart rate variability of the surgical team during laparoscopic and conventional sigmoid resection" Arch Surg (2001) 10.1001/archsurg.136.3.305

[46]

Wadhera "Is the ‘sterile cockpit’ concept applicable to cardiovascular surgery critical intervals or critical events? The impact of protocol-driven communication during cardiopulmonary bypass" J Thorac Cardiovasc Surg (2010) 10.1016/j.jtcvs.2009.10.048

[47]

Wheelock "The impact of operating room distractions on stress, workload, and teamwork" Ann Surg (2015) 10.1097/sla.0000000000001051

[48]

Tiferes "The loud surgeon behind the console: understanding team activities during robot-assisted surgery" J Surg Educ (2016) 10.1016/j.jsurg.2015.12.009

[49]

Andersen "Retention of mastoidectomy skills after virtual reality simulation training" JAMA Otolaryngol Head Neck Surg (2016) 10.1001/jamaoto.2016.0454

[50]

Andersen "Cognitive load in mastoidectomy skills training: virtual reality simulation and traditional dissection compared" J Surg Educ (2016) 10.1016/j.jsurg.2015.09.010

Showing 50 of 118 references

Cited By

205

A multimodal approach combining tool-pressure and EEG features for laparoscopic skill classification using machine learning

Sebahat Selin Sahin, Cagri Zengin · 2026

Artificial Intelligence Surgery

Validation of the French SURG-TLX: A reliable and task-sensitive tool for assessing surgical workload

Georges Pfister, Emeric Saguin · 2026

Journal of Visceral Surgery

Non-Invasive Assessment of Continuous Real-Time Workplace Stress Among Surgeons and Surgical Trainees: A Scoping Review

Egide Abahuje, Katherine A. Lin · 2026

Journal of Surgical Education

Workflow disruptions and surgeon workload: A mixed methods study of physiologically assessed workload and qualitative observation

Liam Wietzorrek, Eugenie Craenen · 2026

Human Factors in Healthcare

Data-Driven Metrics of Operative Performance: The Technology-Enhanced Surgical Training (TEST) Delphi Consensus Study

Lachlan Dick, Victoria Ruth Tallentire · 2025

Journal of Surgical Education

Surgical Sabermetrics

Emma E. Howie, Olivia Ambler · 2024

Annals of Surgery

Cognitive ergonomics and robotic surgery

Shing Wai Wong, Philip Crowe · 2024

Journal of Robotic Surgery

Cognitive Load Management: An Invaluable Tool for Safe and Effective Surgical Training

Emma E Howie, Harini Dharanikota · 2023

Journal of Surgical Education

Demands of surgical teams in robotic-assisted surgery: An assessment of intraoperative workload within different surgical specialties

Jennifer Zamudio, Jeffrey Woodward · 2023

The American Journal of Surgery

Are maladaptive brain changes the reason for burnout and medical error?

Rizwan A. Manji, Jacqueline S. Manji · 2021

The Journal of Thoracic and Cardiov...

Psychometric Properties of NASA-TLX and Index of Cognitive Activity as Measures of Cognitive Workload in Older Adults

Hannes Devos, Kathleen Gustafson · 2020

Brain Sciences

Surgery Task Load Index in Cardiac Surgery: Measuring Cognitive Load Among Teams

Lauren R. Kennedy-Metz, Hill L. Wolfe · 2020

Surgical Innovation

Physiological synchronization and entropy as measures of team cognitive load

Roger D. Dias, Marco A. Zenati · 2019

Journal of Biomedical Informatics

Metrics

205

Citations

118

References

Details

Published: Feb 21, 2018
Vol/Issue: 105(5)
Pages: 491-501
License: View

Authors

R

R D Dias

STRATUS Center for Medical Simulation, Brigham and Women's Hospital, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA

M

M C Ngo-Howard

Department of Otolaryngology – Head and Neck Surgery, Boston University School of Medicine, Boston, Massachusetts, USA; Medical Robotics and Computer Assisted Surgery Laboratory, Division of Cardiac Surgery, Veterans Affairs Boston Healthcare System, West Roxbury, Massachusetts, USA

M

M T Boskovski

Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA

M

M A Zenati

Harvard Medical School, Boston, Massachusetts, USA; Medical Robotics and Computer Assisted Surgery Laboratory, Division of Cardiac Surgery, Veterans Affairs Boston Healthcare System, West Roxbury, Massachusetts, USA

S

S J Yule

STRATUS Center for Medical Simulation, Brigham and Women's Hospital, Boston, Massachusetts, USA; Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA; Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA

Funding

National Heart, Lung, and Blood Institute Award: R01HL126896-01A

Cite This Article

R D Dias, M C Ngo-Howard, M T Boskovski, et al. (2018). Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload. British Journal of Surgery, 105(5), 491-501. https://doi.org/10.1002/bjs.10795

Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

You May Also Like