Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations

Steven Yule; A Gupta; D Gazarian; A Geraghty; D S Smink; J Beard; T Sundt; G Youngson; C McIlhenny; S Paterson-Brown

doi:10.1002/bjs.10779

journal article Mar 30, 2018

Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations

Steven Yule

A Gupta D Gazarian A Geraghty D S Smink J Beard T Sundt G Youngson C McIlhenny S Paterson-Brown

British Journal of Surgery Vol. 105 No. 6 pp. 719-727 · Oxford University Press (OUP)

View at Publisher Save 10.1002/bjs.10779

Abstract

Abstract

Background
Surgeons' non-technical skills are an important part of surgical performance and surgical education. The most widely adopted assessment tool is the Non-Technical Skills for Surgeons (NOTSS) behaviour rating system. Psychometric analysis of this tool to date has focused on inter-rater reliability and feasibility rather than validation.

Methods
NOTSS assessments were collected from two groups of consultant/attending surgeons in the UK and USA, who rated behaviours of the lead surgeon during a video-based simulated crisis scenario after either online or classroom instruction. The process of validation consisted of assessing construct validity, scale reliability and concurrent criterion validity, and undertaking a sensitivity analysis. Central to this was confirmatory factor analysis to evaluate the structure of the NOTSS taxonomy.

Results
Some 255 consultant surgeons participated in the study. The four-category NOTSS model was found to have robust construct validity evidence, and a superior fit compared with alternative models. Logistic regression and sensitivity analysis revealed that, after adjusting for technical skills, for every 1-point increase in NOTSS score of the lead surgeon, the odds of having a higher versus lower patient safety score was 2·29 times. The same pattern of results was obtained for a broad mix of surgical specialties (UK) as well as a single discipline (cardiothoracic, USA).

Conclusion
The NOTSS tool can be applied in research and education settings to measure non-technical skills in a valid and efficient manner.

Topics

No keywords indexed for this article. Browse by subject →

References

45

[1]

Gawande "Analysis of errors reported by surgeons at three teaching hospitals" Surgery (2003) 10.1067/msy.2003.169

[2]

A prospective study of patient safety in the operating room

Caprice K. Christian, Michael L. Gustafson, Emilie M. Roth et al.

Surgery 2006 10.1016/j.surg.2005.07.037

[3]

Non-technical skills for surgeons in the operating room: A review of the literature

S. Yule, R. Flin, S. Paterson-Brown et al.

Surgery 2006 10.1016/j.surg.2005.06.017

[4]

Henrickson Parker "Factors influencing surgeons' intraoperative leadership: analysis of unanticipated events in the operating room" World J Surg (2014) 10.1007/s00268-013-2241-0

[5]

Mazzocco "Surgical team behaviors and patient outcomes" Am J Surg (2009) 10.1016/j.amjsurg.2008.03.002

[6]

Yule "Surgeons' non-technical skills" Surg Clin North Am (2012) 10.1016/j.suc.2011.11.004

[7]

Robertson "Oxford NOTECHS II: a modified theatre team non-technical skills scoring system" PLOS One (2014) 10.1371/journal.pone.0090320

[8]

Development of a rating system for surgeons' non‐technical skills

S Yule, R Flin, S Paterson‐Brown et al.

Medical Education 2006 10.1111/j.1365-2929.2006.02610.x

[9]

Russ "Observational teamwork assessment for surgery: feasibility of clinical and nonclinical assessor calibration with short-term training" Ann Surg (2012) 10.1097/sla.0b013e31824a9a02

[10]

Catchpole "A multicenter trial of aviation-style training for surgical teams" J Patient Saf (2010) 10.1097/pts.0b013e3181f100ea

[11]

Musson "Team training and resource management in health care: current issues and future directions" Harvard Health Policy Rev (2004)

[12]

Hull "The impact of nontechnical skills on technical performance in surgery: a systematic review" J Am Coll Surg (2012) 10.1016/j.jamcollsurg.2011.10.016

[13]

Mete "Estimating the reliability of nontechnical skills in medical teams" J Surg Educ (2017) 10.1016/j.jsurg.2016.12.011

[14]

Beard "Assessing the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology" Health Technol Assess (2011) 10.3310/hta15010

[15]

Scott "The new ACS/APDS Skills Curriculum: moving the learning curve out of the operating room" J Gastrointest Surg (2008) 10.1007/s11605-007-0357-y

[16]

Hull "Delphi Expert Consensus Panel. Training faculty in nontechnical skill assessment: national guidelines on program requirements" Ann Surg (2013) 10.1097/sla.0b013e318279560b

[17]

Crossley "Prospective observational study to evaluate NOTSS (Non-Technical Skills for Surgeons) for assessing trainees' non-technical performance in the operating theatre" Br J Surg (2011) 10.1002/bjs.7478

[18]

Yule "Surgeons' non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system" World J Surg (2008) 10.1007/s00268-007-9320-z

[19]

Wood "Training tools for nontechnical skills for surgeons – a systematic review" J Surg Educ (2017) 10.1016/j.jsurg.2016.11.017

[20]

Flin (2006)

[21]

Yule "Coaching non-technical skills improves surgical residents' performance in a simulated operating room" J Surg Educ (2015) 10.1016/j.jsurg.2015.06.012

[22]

Doumouras "Non-technical skills of surgeons and anaesthetists in simulated operating theatre crises" Br J Surg (2017) 10.1002/bjs.10526

[23]

Tsuburaya "Introduction of the Non-Technical Skills for Surgeons (NOTSS) system in a Japanese cancer center" Surg Today (2016) 10.1007/s00595-016-1322-8

[24]

Spanager "Customization of a tool to assess Danish surgeons' non-technical skills in the operating room" Dan Med J (2012)

[25]

Scott "Contextual challenges to safe surgery in a resource-limited setting: a multicenter, multiprofessional qualitative study" Ann Surg (2017)

[26]

Sharma "Non-technical skills assessment in surgery" Surg Oncol (2011) 10.1016/j.suronc.2010.10.001

[27]

Byrne (1994)

[28]

Gerbing "An updated paradigm for scale development incorporating unidimensionality and its assessment" J Mark Res (1988) 10.1177/002224378802500207

[29]

Neal "study of the lagged relationships among safety climate, safety motivation, safety behavior, and accidents at the individual and group levels" J Appl Psychol (2006) 10.1037/0021-9010.91.4.946

[30]

Zohar "A multilevel model of safety climate: cross-level relationships between organization and group-level climates" J Appl Psychol (2005) 10.1037/0021-9010.90.4.616

[31]

Barling "Development and test of a model linking safety-specific transformational leadership and occupational safety" J Appl Psychol (2002) 10.1037/0021-9010.87.3.488

[32]

Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives

Li‐tze Hu, Peter M. Bentler

Structural Equation Modeling: A Multidisciplinary... 1999 10.1080/10705519909540118

[33]

Pena "Nontechnical skills training for the operating room: a prospective study using simulation and didactic workshop" Surgery (2015) 10.1016/j.surg.2015.02.008

[34]

Maran (2013) 10.1002/9781118748039.ch8

[35]

Caskey "Integration of hands-on team training into existing curriculum improves both technical and nontechnical skills in laparoscopic cholecystectomy" J Surg Educ (2017) 10.1016/j.jsurg.2017.05.007

[36]

Phitayakorn "The relationship between intraoperative teamwork and management skills in patient care" Surgery (2015) 10.1016/j.surg.2015.03.031

[37]

Brunckhorst "Simulation-based ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial" Surg Endosc (2015) 10.1007/s00464-014-3996-6

[38]

Rao "Can simulated team tasks be used to improve nontechnical skills in the operating room?" J Surg Educ (2016) 10.1016/j.jsurg.2016.06.004

[39]

Dedy "Teaching nontechnical skills in surgical residency: a systematic review of current approaches and outcomes" Surgery (2013) 10.1016/j.surg.2013.04.034

[40]

Doumouras "Early crisis nontechnical skill teaching in residency leads to long-term skill retention and improved performance during crises: a prospective, nonrandomized controlled study" Surgery (2017) 10.1016/j.surg.2016.11.022

[41]

Gostlow "Non-technical skills of surgical trainees and experienced surgeons" Br J Surg (2017) 10.1002/bjs.10493

[42]

NHS England National Safety Standards for Invasive Procedures (NatSSIPs) (2015)

[43]

Yule "Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system" ANZ J Surg (2009) 10.1111/j.1445-2197.2008.04833.x

[44]

Alenius "Staffing and resource adequacy strongly related to RNs' assessment of patient safety: a national study of RNs working in acute-care hospitals in Sweden" BMJ Qual Saf (2014) 10.1136/bmjqs-2012-001734

[45]

Szasz "Assessing technical competence in surgical trainees: a systematic review" Ann Surg (2015) 10.1097/sla.0000000000000866

Cited By

69

Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Rayan Ebnali Harari, Roger D. Dias · 2024

JAMA Network Open

Observational Tools That Quantify Nontechnical Skills in the Operating Room: A Systematic Review

Ryan D. McMullan, Rachel Urwin · 2020

Journal of Surgical Research

Non‐technical skills in paramedicine: A scoping review

Ryan Bennett, Niall Mehmed · 2020

Nursing & Health Sciences

Metrics

69

Citations

45

References

Details

Published: Mar 30, 2018
Vol/Issue: 105(6)
Pages: 719-727
License: View

Authors

S

Steven Yule

STRATUS Center for Medical Simulation, Brigham and Women's Hospital, Boston, Massachusetts, USA; Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA; Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA

A

A Gupta

Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA

D

D Gazarian

STRATUS Center for Medical Simulation, Brigham and Women's Hospital, Boston, Massachusetts, USA

A

A Geraghty

Scottish Centre for Simulation and Clinical Human Factors, Larbert, UK

D

D S Smink

Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA; Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA

J

J Beard

Faculty of Medicine, University of Sheffield, Sheffield, UK

T

T Sundt

Harvard Medical School, Boston, Massachusetts, USA; Division of Cardiac Surgery, Corrigan Minehan Heart Center, Massachusetts General Hospital, Boston, Massachusetts, USA

G

G Youngson

Department of Paediatric Surgery, Royal Aberdeen Children's Hospital, School of Medicine, University of Aberdeen, Aberdeen, UK

C

C McIlhenny

Scottish Centre for Simulation and Clinical Human Factors, Larbert, UK

S

S Paterson-Brown

Department of Surgery, Royal Infirmary of Edinburgh, Edinburgh, UK

Cite This Article

Steven Yule, A Gupta, D Gazarian, et al. (2018). Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations. British Journal of Surgery, 105(6), 719-727. https://doi.org/10.1002/bjs.10779

Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations

You May Also Like