Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Joseph K. Goodman; Cynthia E. Cryder; Amar Cheema

doi:10.1002/bdm.1753

journal article Apr 02, 2012

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Joseph K. Goodman Cynthia E. Cryder Amar Cheema

Journal of Behavioral Decision Making Vol. 26 No. 3 pp. 213-224 · Wiley

View at Publisher Save 10.1002/bdm.1753

Abstract

ABSTRACTMechanical Turk (MTurk), an online labor system run by Amazon.com, provides quick, easy, and inexpensive access to online research participants. As use of MTurk has grown, so have questions from behavioral researchers about its participants, reliability, and low compensation. In this article, we review recent research about MTurk and compare MTurk participants with community and student samples on a set of personality dimensions and classic decision‐making biases. Across two studies, we find many similarities between MTurk participants and traditional samples, but we also find important differences. For instance, MTurk participants are less likely to pay attention to experimental materials, reducing statistical power. They are more likely to use the Internet to find answers, even with no incentive for correct responses. MTurk participants have attitudes about money that are different from a community sample's attitudes but similar to students' attitudes. Finally, MTurk participants are less extraverted and have lower self‐esteem than other participants, presenting challenges for some research domains. Despite these differences, MTurk participants produce reliable results consistent with standard decision‐making biases: they are present biased, risk‐averse for gains, risk‐seeking for losses, show delay/expedite asymmetries, and show the certainty effect—with almost no significant differences in effect sizes from other samples. We conclude that MTurk offers a highly valuable opportunity for data collection and recommend that researchers using MTurk (1) include screening questions that gauge attention and language comprehension; (2) avoid questions with factual answers; and (3) consider how individual differences in financial and social domains may influence results. Copyright © 2012 John Wiley & Sons, Ltd.

Topics

No keywords indexed for this article. Browse by subject →

References

38

[1]

Small feedback‐based decisions and their limited correspondence to description‐based decisions

Greg Barron, Ido Erev

Journal of Behavioral Decision Making 10.1002/bdm.443

[2]

10.1177/1745691610393980

[3]

10.1002/(sici)1099-0771(200001/03)13:1<61::aid-bdm343>3.0.co;2-j

[4]

Cryder C. &Loewenstein G.(2010).The time versus money scale. Unpublished data Olin Business School Washington University in St. Louis.

[5]

Eriksson K. "Emotional reactions to losing explain gender differences in entering a risky lottery" Judgment and Decision Making (2010) 10.1017/s1930297500001030

[6]

10.1257/089533005775196732

[7]

Friedman T. L. (2005)

[8]

10.1016/s0092-6566(03)00046-1

[9]

10.1111/j.0956-7976.2004.00715.x

[10]

10.1007/s10683-011-9273-9

[11]

Ipeirotis P.(2010).Demographics of Mechanical Turk. (CeDER Working Paper‐10‐01). New York University. Retrieved fromhttp://hdl.handle.net/2451/29585.

[12]

John O. P. (1999)

[13]

Prospect Theory: An Analysis of Decision under Risk

Daniel Kahneman, Amos Tversky

Econometrica 10.2307/1914185

[14]

10.1002/bdm.433

[15]

10.1287/mnsc.34.2.200

[16]

10.1086/208919

[17]

10.1509/jmkr.43.4.618

[18]

10.3758/s13428-011-0124-6

[19]

10.1509/jmkr.45.6.633

[20]

Nenkov G. "A short form of the maximization scale: Factor structure, reliability, and validity studies" Judgment and Decision Making (2008) 10.1017/s1930297500000395

[21]

10.1016/j.jesp.2009.03.009

[22]

10.1016/j.cognition.2006.12.008

[23]

Running experiments on Amazon Mechanical Turk

Gabriele Paolacci, Jesse Chandler, Panagiotis G. Ipeirotis

Judgment and Decision Making 10.1017/s1930297500002205

[24]

Decision‐making competence: External validation through an individual‐differences approach

Andrew M. Parker, Baruch Fischhoff

Journal of Behavioral Decision Making 10.1002/bdm.481

[25]

10.1086/323732

[26]

10.1016/j.jtbi.2011.03.004

[27]

10.1086/383436

[28]

10.1086/523285

[29]

Measuring Global Self-Esteem: Construct Validation of a Single-Item Measure and the Rosenberg Self-Esteem Scale

Richard W. Robins, Holly M. Hendin, Kali H. Trzesniewski

Personality and Social Psychology Bulletin 10.1177/0146167201272002

[30]

Rosenthal R. (2000)

[31]

10.1177/0146167211398138

[32]

10.1371/journal.pone.0016836

[33]

10.1016/0165-1765(81)90067-7

[34]

10.1037/0022-3514.84.1.205

[35]

Tversky A. "Judgment under uncertainty: Heuristics and biases" Science (1974)

[36]

The Framing of Decisions and the Psychology of Choice

Amos Tversky, Daniel Kahneman

Science 10.1126/science.7455683

[37]

10.1007/bf00122574

[38]

10.1177/0092070399273005

Cited By

1,717

Tipping pressure as a societal stressor: Cognitive and emotional pathways to mastery experience

Mansik Yun · 2026

Applied Psychology

Business Groups after Incidents of Wrongdoing: Exploring the Effectiveness of Differentiated Versus Aligned Impression Management Tactics

Josh Wei‐Jun Hsueh, Melanie Richards · 2025

Journal of Management Studies

Understanding the individual labor supply and wages on digital labor platforms: A microworker perspective

Ling Jiang, Xuefei (Nancy) Deng · 2024

International Journal of Informatio...

Do ride-hailing drivers' psychological behaviors influence operational performance?

Yavuz Idug, Suman Niranjan · 2023

International Journal of Operations...

Resistance to the sharing economy: Why some consumers and providers do not participate in P2P sharing

Tin Huynh, Sebastian Gurtner · 2023

Journal of Cleaner Production

When Information is Missing: Booking Decisions in Peer-To-Peer Accommodation

Yang Ding · 2023

Journal of Quality Assurance in Hos...

Measuring pro-environmental orientation: Testing and building scales

Aaron C. Sparks, Phillip J. Ehret · 2022

Journal of Environmental Psychology

The potential of remote XR experimentation: Defining benefits and limitations through expert survey and case study

Jack Ratcliffe, Laurissa Tokarchuk · 2022

Frontiers in Computer Science

Consumer acceptance of cultured meat: an empirical analysis of the role of organizational factors

Nick Lin-Hi, Marlene Reimer · 2022

Journal of Business Economics

Coping with workplace sexual harassment: Social media as an empowered outcome

Akanksha Malik, Shuchi Sinha · 2022

Journal of Business Research

Artificial intelligence in public services: When and why citizens accept its usage

Tanja Sophie Gesk, Michael Leyer · 2022

Government Information Quarterly

Evaluating CloudResearch’s Approved Group as a solution for problematic data quality on MTurk

David J. Hauser, Aaron J. Moss · 2022

Behavior Research Methods

When Managers Become Robin Hoods: A Mixed Method Investigation

Russell Cropanzano, Daniel P. Skarlicki · 2021

Business Ethics Quarterly

Concerns and recommendations for using AmazonMTurkfor eating disorder research

C. Blair Burnette, Jessica L. Luzier · 2021

International Journal of Eating Dis...

Ambient lighting, use of outdoor spaces and perceptions of public safety: evidence from a survey experiment

Jacob Kaplan, Aaron Chalfin · 2021

Security Journal

Data quality of platforms and panels for online behavioral research

Eyal Peer, David Rothschild · 2021

Behavior Research Methods

Using social learning theory in examining YouTube viewers’ desire to imitate travel vloggers

Linh Ha Le, Murat Hancer · 2021

Journal of Hospitality and Tourism...

Combating misinformation: The effects of infographics in verifying false vaccine news

Shawn Domgaard, Mina Park · 2021

Health Education Journal

How do authenticity and quality perceptions affect dining experiences and recommendations of food trucks? The moderating role of perceived risk

Kiyan Shafieizadeh, Salman Alotaibi · 2021

International Journal of Hospitalit...

To support or to boycott: a public segmentation model in corporate social advocacy

Cheng Hong, Cong Li · 2020

Journal of Public Relations Researc...

Metrics

1,717

Citations

38

References

Details

Published: Apr 02, 2012
Vol/Issue: 26(3)
Pages: 213-224
License: View

Authors

J

Joseph K. Goodman

Olin Business School Washington University in St. Louis Saint Louis MO USA

C

Cynthia E. Cryder

Olin Business School Washington University in St. Louis Saint Louis MO USA

A

Amar Cheema

McIntire School of Commerce University of Virginia Charlottesville VA USA

Cite This Article

Joseph K. Goodman, Cynthia E. Cryder, Amar Cheema (2012). Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making, 26(3), 213-224. https://doi.org/10.1002/bdm.1753

Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

You May Also Like