Methodological reporting behavior, sample sizes, and statistical power in studies of event‐related potentials: Barriers to reproducibility and replicability
Methodological reporting guidelines for studies of ERPs were updated in
Psychophysiology
in 2014. These guidelines facilitate the communication of key methodological parameters (e.g., preprocessing steps). Failing to report key parameters represents a barrier to replication efforts, and difficulty with replicability increases in the presence of small sample sizes and low statistical power. We assessed whether guidelines are followed and estimated the average sample size and power in recent research. Reporting behavior, sample sizes, and statistical designs were coded for 150 randomly sampled articles from five high‐impact journals that frequently published ERP research from 2011 to 2017. An average of 63% of guidelines were reported, and reporting behavior was similar across journals, suggesting that gaps in reporting is a shortcoming of the field rather than any specific journal. Publication of the guidelines article had no impact on reporting behavior, suggesting that editors and peer reviewers are not enforcing these recommendations. The average sample size per group was 21. Statistical power was conservatively estimated as .72‒.98 for a large effect size, .35‒.73 for a medium effect, and .10‒.18 for a small effect. These findings indicate that failing to report key guidelines is ubiquitous and that ERP studies are primarily powered to detect large effects. Such low power and insufficient following of reporting guidelines represent substantial barriers to replication efforts. The methodological transparency and replicability of studies can be improved by the open sharing of processing code and experimental tasks and by a priori sample size calculations to ensure adequately powered studies.
No keywords indexed for this article. Browse by subject →
Katherine S. Button, John P. A. Ioannidis, Claire Mokrysz et al.
Iain Chalmers, Paul Glasziou
Franz Faul, Edgar Erdfelder, Axel Buchner et al.
John P. A. Ioannidis
John P. A. Ioannidis
Daniël Lakens
Steven J. Luck, Nicholas Gaspelin
Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop et al.
Showing 50 of 75 references
- Published
- Jul 19, 2019
- Vol/Issue
- 56(11)
- License
- View
You May Also Like
James J. Gross · 2002
3,323 citations
Gary G. Berntson, J. THOMAS BIGGER · 1997
2,977 citations
Risto Näätänen, Terence Picton · 1987
2,660 citations
Margaret M. Bradley, Laura Miccoli · 2008
1,539 citations
Alan J. Fridlund, John T. Cacioppo · 1986
1,303 citations