The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

Yaakov Ophir; Itay Sisso; Christa S.C. Asterhan; Refael Tikochinski; Roi Reichart

doi:10.1177/2167702619865973

The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

Yaakov Ophir^*, Itay Sisso, Christa S.C. Asterhan, Refael Tikochinski, Roi Reichart

^*Corresponding author for this work

Department of Education

Research output: Contribution to journal › Article › peer-review

74 Scopus citations

Abstract

Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided.

Original language	American English
Pages (from-to)	65-83
Number of pages	19
Journal	Clinical Psychological Science
Volume	8
Issue number	1
DOIs	https://doi.org/10.1177/2167702619865973
State	Published - 1 Jan 2020

Bibliographical note

Funding Information:
https://orcid.org/0000-0002-1598-2559 Ophir Yaakov 1 2 Sisso Itay 3 Asterhan Christa S. C. 1 Tikochinski Refael 1 Reichart Roi 2 1 School of Education, The Hebrew University of Jerusalem 2 Industrial Engineering and Management, Technion–Israel Institute of Technology 3 Cognitive Science, The Hebrew University of Jerusalem Yaakov Ophir, School of Education, Hebrew University of Jerusalem, Mount Scopus, Jerusalem, Israel, 91905 E-mail: yaakov.ophir@mail.huji.ac.il 9 2019 2167702619865973 7 1 2019 4 6 2019 © The Author(s) 2019 2019 Association for Psychological Science Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided. depression crowdsourcing Mechanical Turk prevalence of depression data quality measures open data open materials Israel Innovation Authority KAMIN 60560. 60561 special-property open-data special-property open-materials edited-state corrected-proof Action Editor Kelly L. Klump served as action editor for this article. Author Contributions Y. Ophir and I. Sisso developed the study concept. All of the authors contributed to the study design. Test battery development and data collection were performed by I. Sisso and Y. Ophir. I. Sisso performed the data analyses with help from Y. Ophir. Y. Ophir drafted the manuscript, and C. S. C. Asterhan, I. Sisso, and R. Reichart provided critical revisions. The research was conducted under the supervision of C. S. C. Asterhan and R. Reichart. Y. Ophir and It. Sisso contributed equally to this research. All of the authors approved the final manuscript for submission. ORCID iD Yaakov Ophir https://orcid.org/0000-0002-1598-2559 Declaration of Conflicting Interests The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article. Funding The research presented here was conducted with the financial support of the Israeli Innovation Authority in the Ministry of Economy (“Kamin” Grants 60561 and 60560). Supplemental Material Additional supporting information can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 Open Practices All data and materials have been made publicly available via Open Science Framework and can be accessed at https://osf.io/u8ctz . The complete Open Practices Disclosure for this article can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 . This article has received badges for Open Data and Open Materials. More information about the Open Practices badges can be found at https://www.psychologicalscience.org/publications/badges .

Publisher Copyright:
© The Author(s) 2019.

Keywords

Mechanical Turk
crowdsourcing
data quality measures
depression
open data
open materials
prevalence of depression

Access to Document

10.1177/2167702619865973

Cite this

@article{606534b46f664d65af60e2171c056008,

title = "The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon{\textquoteright}s Mechanical Turkers",

abstract = "Data collection from online platforms, such as Amazon{\textquoteright}s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided.",

keywords = "Mechanical Turk, crowdsourcing, data quality measures, depression, open data, open materials, prevalence of depression",

author = "Yaakov Ophir and Itay Sisso and Asterhan, {Christa S.C.} and Refael Tikochinski and Roi Reichart",

note = "Funding Information: https://orcid.org/0000-0002-1598-2559 Ophir Yaakov 1 2 Sisso Itay 3 Asterhan Christa S. C. 1 Tikochinski Refael 1 Reichart Roi 2 1 School of Education, The Hebrew University of Jerusalem 2 Industrial Engineering and Management, Technion–Israel Institute of Technology 3 Cognitive Science, The Hebrew University of Jerusalem Yaakov Ophir, School of Education, Hebrew University of Jerusalem, Mount Scopus, Jerusalem, Israel, 91905 E-mail: yaakov.ophir@mail.huji.ac.il 9 2019 2167702619865973 7 1 2019 4 6 2019 {\textcopyright} The Author(s) 2019 2019 Association for Psychological Science Data collection from online platforms, such as Amazon{\textquoteright}s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided. depression crowdsourcing Mechanical Turk prevalence of depression data quality measures open data open materials Israel Innovation Authority KAMIN 60560. 60561 special-property open-data special-property open-materials edited-state corrected-proof Action Editor Kelly L. Klump served as action editor for this article. Author Contributions Y. Ophir and I. Sisso developed the study concept. All of the authors contributed to the study design. Test battery development and data collection were performed by I. Sisso and Y. Ophir. I. Sisso performed the data analyses with help from Y. Ophir. Y. Ophir drafted the manuscript, and C. S. C. Asterhan, I. Sisso, and R. Reichart provided critical revisions. The research was conducted under the supervision of C. S. C. Asterhan and R. Reichart. Y. Ophir and It. Sisso contributed equally to this research. All of the authors approved the final manuscript for submission. ORCID iD Yaakov Ophir https://orcid.org/0000-0002-1598-2559 Declaration of Conflicting Interests The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article. Funding The research presented here was conducted with the financial support of the Israeli Innovation Authority in the Ministry of Economy (“Kamin” Grants 60561 and 60560). Supplemental Material Additional supporting information can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 Open Practices All data and materials have been made publicly available via Open Science Framework and can be accessed at https://osf.io/u8ctz . The complete Open Practices Disclosure for this article can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 . This article has received badges for Open Data and Open Materials. More information about the Open Practices badges can be found at https://www.psychologicalscience.org/publications/badges . Publisher Copyright: {\textcopyright} The Author(s) 2019.",

year = "2020",

month = jan,

day = "1",

doi = "10.1177/2167702619865973",

language = "American English",

volume = "8",

pages = "65--83",

journal = "Clinical Psychological Science",

issn = "2167-7026",

publisher = "SAGE Publications Inc.",

number = "1",

}

TY - JOUR

T1 - The Turker Blues

T2 - Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

AU - Ophir, Yaakov

AU - Sisso, Itay

AU - Asterhan, Christa S.C.

AU - Tikochinski, Refael

AU - Reichart, Roi

N1 - Funding Information: https://orcid.org/0000-0002-1598-2559 Ophir Yaakov 1 2 Sisso Itay 3 Asterhan Christa S. C. 1 Tikochinski Refael 1 Reichart Roi 2 1 School of Education, The Hebrew University of Jerusalem 2 Industrial Engineering and Management, Technion–Israel Institute of Technology 3 Cognitive Science, The Hebrew University of Jerusalem Yaakov Ophir, School of Education, Hebrew University of Jerusalem, Mount Scopus, Jerusalem, Israel, 91905 E-mail: yaakov.ophir@mail.huji.ac.il 9 2019 2167702619865973 7 1 2019 4 6 2019 © The Author(s) 2019 2019 Association for Psychological Science Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided. depression crowdsourcing Mechanical Turk prevalence of depression data quality measures open data open materials Israel Innovation Authority KAMIN 60560. 60561 special-property open-data special-property open-materials edited-state corrected-proof Action Editor Kelly L. Klump served as action editor for this article. Author Contributions Y. Ophir and I. Sisso developed the study concept. All of the authors contributed to the study design. Test battery development and data collection were performed by I. Sisso and Y. Ophir. I. Sisso performed the data analyses with help from Y. Ophir. Y. Ophir drafted the manuscript, and C. S. C. Asterhan, I. Sisso, and R. Reichart provided critical revisions. The research was conducted under the supervision of C. S. C. Asterhan and R. Reichart. Y. Ophir and It. Sisso contributed equally to this research. All of the authors approved the final manuscript for submission. ORCID iD Yaakov Ophir https://orcid.org/0000-0002-1598-2559 Declaration of Conflicting Interests The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article. Funding The research presented here was conducted with the financial support of the Israeli Innovation Authority in the Ministry of Economy (“Kamin” Grants 60561 and 60560). Supplemental Material Additional supporting information can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 Open Practices All data and materials have been made publicly available via Open Science Framework and can be accessed at https://osf.io/u8ctz . The complete Open Practices Disclosure for this article can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619865973 . This article has received badges for Open Data and Open Materials. More information about the Open Practices badges can be found at https://www.psychologicalscience.org/publications/badges . Publisher Copyright: © The Author(s) 2019.

PY - 2020/1/1

Y1 - 2020/1/1

N2 - Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided.

AB - Data collection from online platforms, such as Amazon’s Mechanical Turk (MTurk), has become popular in clinical research. However, there are also concerns about the representativeness and the quality of these data for clinical studies. The present work explores these issues in the specific case of major depression. Analyses of two large data sets gathered from MTurk (Sample 1: N = 2,692; Sample 2: N = 2,354) revealed two major findings: First, failing to screen for inattentive and fake respondents inflates the rates of major depression artificially and significantly (by 18.5%–27.5%). Second, after cleaning the data sets, depression in MTurk is still 1.6 to 3.6 times higher than general population estimates. Approximately half of this difference can be attributed to differences in the composition of MTurk samples and the general population (i.e., sociodemographics, health, and physical activity lifestyle). Several explanations for the other half are proposed, and practical data-quality tools are provided.

KW - Mechanical Turk

KW - crowdsourcing

KW - data quality measures

KW - depression

KW - open data

KW - open materials

KW - prevalence of depression

UR - http://www.scopus.com/inward/record.url?scp=85074608412&partnerID=8YFLogxK

U2 - 10.1177/2167702619865973

DO - 10.1177/2167702619865973

M3 - Article

AN - SCOPUS:85074608412

SN - 2167-7026

VL - 8

SP - 65

EP - 83

JO - Clinical Psychological Science

JF - Clinical Psychological Science

IS - 1

ER -

The Turker Blues: Hidden Factors Behind Increased Depression Rates Among Amazon’s Mechanical Turkers

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this