The impact of peer review on the contribution potential of scientific papers

Akira Matsui; Emily Chen; Yunwen Wang; Emilio Ferrara

doi:10.7717/peerj.11999

journal article Open Access Sep 15, 2021

The impact of peer review on the contribution potential of scientific papers

Akira Matsui Emily Chen

Yunwen Wang

Emilio Ferrara

PeerJ Vol. 9 pp. e11999 · PeerJ

View at Publisher Save 10.7717/peerj.11999

Abstract

The peer-reviewing process has long been regarded as an indispensable tool in ensuring the quality of a scientific publication. While previous studies have tried to understand the process as a whole, not much effort has been devoted to investigating the determinants and impacts of the content of the peer review itself. This study leverages open data from nearly 5,000 PeerJ publications that were eventually accepted. Using sentiment analysis, Latent Dirichlet Allocation (LDA) topic modeling, mixed linear regression models, and logit regression models, we examine how the peer-reviewing process influences the acceptance timeline and contribution potential of manuscripts, and what modifications were typically made to manuscripts prior to publication. In an open review paradigm, our findings indicate that peer reviewers’ choice to reveal their names in lieu of remaining anonymous may be associated with more positive sentiment in their review, implying possible social pressure from name association. We also conduct a taxonomy of the manuscript modifications during a revision, studying the words added in response to peer reviewer feedback. This study provides insights into the content of peer reviews and the subsequent modifications authors make to their manuscripts.

Topics

No keywords indexed for this article. Browse by subject →

References

33

[1]

Bird (2009)

[2]

Blei "Latent dirichlet allocation" Journal of Machine Learning Research (2003)

[3]

Bornmann "Scientific peer review" Annual Review of Information Science and Technology (2011) 10.1002/aris.2011.1440450112

[4]

Bowman "Predatory publishing, questionable peer review, and fraudulent conferences" American Journal of Pharmaceutical Education (2014) 10.5688/ajpe7810176

[5]

Bravo "The effect of publishing peer review reports on referee behavior in five scholarly journals" Nature Communications (2019) 10.1038/s41467-018-08250-2

[6]

Buljan "Meta-research: large-scale language analysis of peer review reports" Elife (2020) 10.7554/elife.53249

[7]

Card "Are referees and editors in economics gender neutral" Quarterly Journal of Economics (2020) 10.1093/qje/qjz035

[8]

Chang "Reading tea leaves: how humans interpret topic models" (2009)

[9]

Clark "Firm action needed on predatory journals" BMJ (2015) 10.1136/bmj.h210

[10]

Djupe "Peer reviewing in political science: new survey results" PS: Political Science & Politics (2015)

[11]

Science of science

Santo Fortunato, Carl T. Bergstrom, Katy Börner et al.

Science 2018 10.1126/science.aao0185

[12]

Golden "Quantifying the volunteer effort of scientific peer reviewing" Bulletin of the American Meteorological Society (2012) 10.1175/bams-d-11-00129.1

[13]

Greene (2003)

[14]

Grimaldo "Fragments of peer review: a quantitative analysis of the literature (1969-2015)" PLOS ONE (2018) 10.1371/journal.pone.0193148

[15]

Grogan "How the entire scientific community can confront gender bias in the workplace" Nature Ecology & Evolution (2019) 10.1038/s41559-018-0747-4

[16]

Harnad (1996)

[17]

Hechtman "Nih funding longevity by gender" Proceedings of the National Academy of Sciences of the United States of America (2018) 10.1073/pnas.1800615115

[18]

Hemlin "The shift in academic quality control" Science, Technology, & Human Values (2006) 10.1177/0162243905283639

[19]

Henderson "Problems with peer review" BMJ (2010) 10.1136/bmj.c1409

[20]

Hirsch "An index to quantify an individual’s scientific research output" Proceedings of the National Academy of Sciences of the United States of America (2005) 10.1073/pnas.0507655102

[21]

Justice "Does masking author identity improve peer review quality?: A randomized controlled trial" JAMA (1998) 10.1001/jama.280.3.240

[22]

Kovanis "The global burden of journal peer review in the biomedical literature: Strong imbalance in the collective enterprise" PLOS ONE (2016) 10.1371/journal.pone.0166387

[23]

Laycock "The influence of first author sex on acceptance rates of submissions to anaesthesia cases" Anaesthesia (2019) 10.1111/anae.14797

[24]

Murray "Author-reviewer homophily in peer review" BioRxiv (2019) 10.1101/400515

[25]

Nielsen "Opinion: gender diversity leads to better science" Proceedings of the National Academy of Sciences of the United States of America (2017) 10.1073/pnas.1700616114

[26]

Orduña-Malea "Dimensions: redescubriendo el ecosistema de la información científica" El Profesional de la Información (2018) 10.3145/epi.2018.mar.21

[27]

Santamaría "Comparison and benchmark of name-to-gender inference services" PeerJ Computer Science (2018) 10.7717/peerj-cs.156

[28]

Schroter "Effects of training on quality of peer review: randomised controlled trial" BMJ (2004) 10.1136/bmj.38023.700775.ae

[29]

Stevens "Exploring topic coherence over many models and many topics" (2012)

[30]

Superchi "Tools used to assess the quality of peer review reports: a methodological systematic review" BMC Medical Research Methodology (2019) 10.1186/s12874-019-0688-x

[31]

Tomkins "Reviewer bias in single-versus double-blind peer review" Proceedings of the National Academy of Sciences of the United States of America (2017) 10.1073/pnas.1707323114

[32]

Wallach "Evaluation methods for topic models" (2009)

[33]

Wicherts "Peer review quality and transparency of the peer-review process in open access and subscription journals" PLOS ONE (2016) 10.1371/journal.pone.0147913

Cited By

20

Textual features of peer review predict top-cited papers: An interpretable machine learning perspective

Zhuanlan Sun · 2024

Journal of Informetrics

The academic status of reviewers predicts their language use

Zhuanlan Sun, C. Clark Cao · 2023

Journal of Informetrics

Understanding the peer review endeavor in scientific publishing

Guangyao Zhang, Shenmeng Xu · 2022

Journal of Informetrics

Metrics

20

Citations

33

References

Details

Published: Sep 15, 2021
Vol/Issue: 9
Pages: e11999
License: View

Authors

A

Akira Matsui

Department of Computer Science, University of Southern California, Los Angeles, California, United States; Information Sciences Institute, University of Southern California, Marina del Rey, California, United States

E

Emily Chen

Department of Computer Science, University of Southern California, Los Angeles, California, United States; Information Sciences Institute, University of Southern California, Marina del Rey, California, United States

Y

Yunwen Wang

Annenberg School for Communication and Journalism, University of Southern California, Los Angeles, California, United States

E

Emilio Ferrara

Department of Computer Science, University of Southern California, Los Angeles, California, United States; Information Sciences Institute, University of Southern California, Marina del Rey, California, United States; Annenberg School for Communication and Journalism, University of Southern California, Los Angeles, California, United States

Cite This Article

Akira Matsui, Emily Chen, Yunwen Wang, et al. (2021). The impact of peer review on the contribution potential of scientific papers. PeerJ, 9, e11999. https://doi.org/10.7717/peerj.11999

The impact of peer review on the contribution potential of scientific papers

You May Also Like