Unveiling variability: A systematic review of reproducibility in visual EEG analysis, with focus on seizures

Eivind Aanestad; Sándor Beniczky; Henning Olberg; Jan Brogger

doi:10.1002/epd2.20291

journal article Open Access Sep 28, 2024

Unveiling variability: A systematic review of reproducibility in visual EEG analysis, with focus on seizures

Eivind Aanestad Sándor Beniczky

Henning Olberg Jan Brogger

Epileptic Disorders Vol. 26 No. 6 pp. 827-839 · Wiley

View at Publisher Save 10.1002/epd2.20291

Abstract

AbstractObjectiveReproducibility is key for diagnostic tests involving subjective evaluation by experts. Our aim was to systematically review the reproducibility of visual analysis in clinical electroencephalogram (EEG). In this paper, we give data on the scope of EEG features found, and detailed reproducibility data for the most studied feature.MethodsWe searched four databases for articles reporting reproducibility in clinical EEG, until June 2023. Two raters screened 24 553 citations, and then 2736 full texts. Quality was assessed according to the GRRAS guidelines.ResultsWe found 275 studies (268 interrater and 20 intrarater), addressing 606 different EEG features. Only 38 EEG features had been studied in >2 studies. Most studies had <50 patients and EEGs. The most often addressed feature was seizure detection (62 papers). Interrater reproducibility of seizure detection was substantial‐to‐almost‐perfect with experienced raters and raw EEG (kappa .62–.88). With experienced raters and transformed EEG, reproducibility was substantial (kappa .63–.70). Inexperienced raters had lower reproducibility. Seizure lateralization reproducibility was moderate to substantial (kappa .58–.77) but lower than for seizure detection.SignificanceMost EEG reproducibility studies are done only once. Intrarater studies are rare. The reproducibility of visual EEG analysis is variable. Interrater reproducibility for seizure detection is substantial‐to‐perfect with experienced raters and raw EEG, less with inexperienced raters or transformed EEG.The results of visual EEG analysis vary within the same rater, and between raters. There is a need for larger collaborative studies, using improved methodology, as well as more intrarater studies of EEG interpretation.

Topics

No keywords indexed for this article. Browse by subject →

References

43

[1]

10.1016/j.clinph.2018.01.019

[2]

10.1891/9781617051807

[3]

10.1001/archneurpsyc.1935.02250240002001

[4]

10.1212/wnl.4.2.143

[5]

10.1016/j.clinph.2021.10.007

[6]

10.1017/cjn.2022.15

[7]

10.1001/jamaneurol.2019.3485

[8]

10.1001/jamaneurol.2023.1645

[9]

10.1148/radiol.211593

[10]

10.1136/bmj.309.6954.597

[11]

10.1016/j.seizure.2015.09.014

[12]

10.1111/ene.13219

[13]

10.1186/s13643-018-0699-4

[14]

Rayyan—a web and mobile app for systematic reviews

Mourad Ouzzani, Hossam Hammady, Zbys Fedorowicz et al.

Systematic Reviews 10.1186/s13643-016-0384-4

[15]

Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed

Jan Kottner, Laurent Audigé, Stig Brorson et al.

Journal of Clinical Epidemiology 10.1016/j.jclinepi.2010.03.002

[16]

10.1542/peds.103.6.1263

[17]

10.1177/088307380101600822

[18]

10.1097/wnp.0000000000000634

[19]

10.1136/adc.2009.165969

[20]

10.1111/j.1528-1157.1998.tb01355.x

[21]

10.1097/wnp.0000000000000534

[22]

10.1016/j.seizure.2008.10.010

[23]

10.1097/wnp.0000000000000812

[24]

10.1016/j.clinph.2017.07.418

[25]

10.1111/epi.16655

[26]

American Clinical Neurophysiology Society's Standardized Critical Care EEG Terminology: 2021 Version

Lawrence J. Hirsch, Michael W.K. Fong, Markus Leitinger et al.

Journal of Clinical Neurophysiology 10.1097/wnp.0000000000000806

[27]

10.1097/wnp.0000000000000424

[28]

Interrater agreement for Critical Care EEG Terminology

Nicolas Gaspard, Lawrence J. Hirsch, Suzette M. LaRoche et al.

Epilepsia 10.1111/epi.12653

[29]

10.1016/j.clinph.2015.03.017

[30]

10.1038/s41598-021-92827-3

[31]

10.1371/journal.pone.0085966

[32]

10.1017/s0317167100051507

[33]

Volavka J "Die Zuverlässigkeit der EEG‐Beurteilung" Z EEG‐EMG (1973)

[34]

10.1046/j.1440-1819.2003.01152.x

[35]

Ramirez‐Campos J "Intra and inter‐observer reliability of electroencephalogram interpretation in pediatric patients with neurological alterations" Rev Med Inst Mex Seguro Soc (2014)

[36]

10.1001/archneur.1992.00530270045017

[37]

AttiaTP RobbinsK BeniczkyS Bosch‐BayardJ DelormeA LundstromBN et al.Hierarchical Event Descriptor library schema for EEG data annotation. arXiv preprint arXiv:2310151732023.

[38]

10.1038/s41597-019-0104-8

[39]

10.1016/j.clinph.2023.07.008

[40]

10.1016/j.yebeh.2014.01.011

[41]

10.1016/j.amjmed.2008.01.001

[42]

Chen P‐HC "Evaluation of artificial intelligence on a reference standard based on subjective interpretation. Lancet Digit" Health (2021)

[43]

Jing J "Interrater reliability of expert Electroencephalographers identifying seizures and rhythmic and periodic patterns in EEGs" Neurology (2023)

Metrics

3

Citations

43

References

Details

Published: Sep 28, 2024
Vol/Issue: 26(6)
Pages: 827-839
License: View

Authors

E

Eivind Aanestad

Department of Clinical Neurophysiology Haukeland University Hospital Bergen Norway; Department of Clinical Medicine University of Bergen Bergen Norway

S

Sándor Beniczky

Danish Epilepsy Centre, Dianalund Denmark and Aarhus University Aarhus Denmark

H

Henning Olberg

Department of Clinical Neurophysiology Haukeland University Hospital Bergen Norway

J

Jan Brogger

Department of Clinical Neurophysiology Haukeland University Hospital Bergen Norway

Funding

Helse Vest Award: F‐10226

Cite This Article

Eivind Aanestad, Sándor Beniczky, Henning Olberg, et al. (2024). Unveiling variability: A systematic review of reproducibility in visual EEG analysis, with focus on seizures. Epileptic Disorders, 26(6), 827-839. https://doi.org/10.1002/epd2.20291

Unveiling variability: A systematic review of reproducibility in visual EEG analysis, with focus on seizures

You May Also Like