Detecting 
            p‐Hacking

Graham Elliott; Nikolay Kudrin; Kaspar Wüthrich

doi:10.3982/ecta18583

journal article Jan 01, 2022

Detecting p‐Hacking

Graham Elliott Nikolay Kudrin Kaspar Wüthrich

Econometrica Vol. 90 No. 2 pp. 887-906 · JSTOR

View at Publisher Save 10.3982/ecta18583

Abstract

We theoretically analyze the problem of testing for
p‐hacking based on distributions of
p‐values across multiple studies. We provide general results for when such distributions have testable restrictions (are non‐increasing) under the null of no
p‐hacking. We find novel additional testable restrictions for
p‐values based on
t‐tests. Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution of
p‐values. These testable restrictions result in more powerful tests for the null hypothesis of no
p‐hacking. When there is also publication bias, our tests are joint tests for
p‐hacking and publication bias. A reanalysis of two prominent data sets shows the usefulness of our new tests.

Topics

No keywords indexed for this article. Browse by subject →

References

33

[1]

Andrews American Economic Review (2019) 10.1257/aer.20180310

[2]

Beare, B. K. (2021): “Least Favorability of the Uniform Distribution for Tests of the Concavity of a Distribution Function,” Stat. e376, https://onlinelibrary.wiley.com/doi/abs/10.1002/sta4.376. 10.1002/sta4.376

[3]

Beare Econometric Theory (2015) 10.1017/s0266466614000401

[4]

Brodeur American Economic Review (2020) 10.1257/aer.20190687

[5]

Brodeur, A., M. Lé, M. Sangnier, and Y. Zylberberg (2016a): “Replication Data for: Star Wars: The Empirics Strike Back,” Nashville, TN: American Economic Association [publisher], 2016. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2019-10-12, https://doi.org/10.3886/E113633V1 (last accessed 09/23/2020).

[6]

Brodeur American Economic Journal: Applied Economics (2016)

[7]

Bruns Research Policy (2019) 10.1016/j.respol.2019.05.005

[8]

Carolan Biometrika (2005) 10.1093/biomet/92.1.159

[9]

Cattaneo Journal of the American Statistical Association (2020) 10.1080/01621459.2019.1635480

[10]

Cattaneo, M. D., M. Jansson, and X. Ma (2021): “rddensity: Manipulation Testing Based on Density Discontinuity,” R package version 2.2.

[11]

Christensen Journal of Economic Literature (2018) 10.1257/jel.20171350

[12]

Cox, G., and X. Shi (2020): “Simple Adaptive Size-Exact Testing for Full-Vector and Subvector Inference in Moment Inequality Models,” arXiv:1907.06317v2.

[13]

Winter PeerJ (2015) 10.7717/peerj.733

[14]

Elliott, G., N. Kudrin, and K. Wüthrich (2020): “Detecting p-Hacking,” arXiv:1906.06711v3.

[15]

Detecting p‐Hacking

Graham Elliott, Nikolay Kudrin, Kaspar Wüthrich

Econometrica 10.3982/ecta18583

[16]

Fang Electronic Journal of Statistics (2019) 10.1214/19-ejs1638

[17]

Gerber Quarterly Journal of Political Science (2008) 10.1561/100.00008024

[18]

The Extent and Consequences of P-Hacking in Science

Megan L. Head, Luke Holman, Rob Lanfear et al.

PLOS Biology 2015 10.1371/journal.pbio.1002106

[19]

Head, M. L., L. Holman, R. Lanfear, A. T. Kahn, and M. D. Jennions (2016): “Data From: The Extent and Consequences of p-Hacking in Science,” Dryad, Dataset, https://datadryad.org/resource/doi:10.5061/dryad.79d43 (last accessed 09/29/2020).

[20]

Hung Biometrics (1997) 10.2307/2533093

[21]

Karunamuni Statistical Methodology (2005) 10.1016/j.stamet.2005.04.001

[22]

Kulikov Journal of Theoretical Probability (2008) 10.1007/s10959-007-0103-0

[23]

Leggett The Quarterly Journal of Experimental Psychology (2013) 10.1080/17470218.2013.863371

[24]

Masicampo The Quarterly Journal of Experimental Psychology (2012) 10.1080/17470218.2012.711335

[25]

McCrary Journal of Econometrics (2008) 10.1016/j.jeconom.2007.05.005

[26]

(2020)

[27]

Simonsohn Journal of Experimental Psychology: General (2014) 10.1037/a0033242

[28]

Simonsohn Journal of Experimental Psychology: General (2015) 10.1037/xge0000104

[29]

Snyder, C., and R. Zhuo (2018): “Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias,” NBER WP 25058. 10.31222/osf.io/8vdrh

[30]

StataCorp (2019)

[31]

Ulrich Journal of Experimental Psychology: General (2015) 10.1037/xge0000086

[32]

Ulrich Psychological Methods (2018) 10.1037/met0000125

[33]

Vivalt Oxford Bulletin of Economics and Statistics (2019) 10.1111/obes.12289

Cited By

53

Relative Risk Aversion: A Meta‐Analysis

Ali Elminejad, Tomas Havranek · 2025