The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

Yaakov Ophir*, Itay Sisso, Christa S.C. Asterhan, Refael Tikochinski, Roi Reichart

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

74 Scopus citations

Abstract

Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided.

Original languageAmerican English
Pages (from-to)65-83
Number of pages19
JournalClinical Psychological Science
Volume8
Issue number1
DOIs
StatePublished - 1 Jan 2020

Bibliographical note

Funding Information:
https://orcid.org/0000-0002-1598-2559 Ophir Yaakov 1 2 Sisso Itay 3 Asterhan Christa S. C. 1 Tikochinski Refael 1 Reichart Roi 2 1 School of Education, The Hebrew University of Jerusalem 2 Industrial Engineering and Management, Technion–Israel Institute of Technology 3 Cognitive Science, The Hebrew University of Jerusalem Yaakov Ophir, School of Education, Hebrew University of Jerusalem, Mount Scopus, Jerusalem, Israel, 91905 E-mail: yaakov.ophir@mail.huji.ac.il 9 2019 2167702619865973 7 1 2019 4 6 2019 © The Author(s) 2019 2019 Association for Psychological Science Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided. depression crowdsourcing Mechanical Turk prevalence of depression data quality measures open data open materials Israel Innovation Authority KAMIN 60560. 60561 special-property open-data special-property open-materials edited-state corrected-proof Action Editor Kelly L. Klump served as action editor for this article. Author Contributions Y. Ophir and I. Sisso developed the study concept. All of the authors contributed to the study design. Test battery development and data collection were performed by I. Sisso and Y. Ophir. I. Sisso performed the data analyses with help from Y. Ophir. Y. Ophir drafted the manuscript, and C. S. C. Asterhan, I. Sisso, and R. Reichart provided critical revisions. The research was conducted under the supervision of C. S. C. Asterhan and R. Reichart. Y. Ophir and It. Sisso contributed equally to this research. All of the authors approved the final manuscript for submission. ORCID iD Yaakov Ophir https://orcid.org/0000-0002-1598-2559 Declaration of Conflicting Interests The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article. Funding The research presented here was conducted with the financial support of the Israeli Innovation Authority in the Ministry of Economy (“Kamin” Grants 60561 and 60560). Supplemental Material Additional supporting information can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 Open Practices All data and materials have been made publicly available via Open Science Framework and can be accessed at https://osf.io/u8ctz . The complete Open Practices Disclosure for this article can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 . This article has received badges for Open Data and Open Materials. More information about the Open Practices badges can be found at https://www.psychologicalscience.org/publications/badges .

Publisher Copyright:
© The Author(s) 2019.

Keywords

  • Mechanical Turk
  • crowdsourcing
  • data quality measures
  • depression
  • open data
  • open materials
  • prevalence of depression

Fingerprint

Dive into the research topics of 'The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers'. Together they form a unique fingerprint.

Cite this