Ballpark crowdsourcing: The wisdom of rough group comparisons

Tom Hope, Dafna Shahaf

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

Crowdsourcing has become a popular method for collecting labeled training data. However, in many practical scenarios traditional labeling can be difficult for crowdworkers (for example, if the data is high-dimensional or unintuitive, or the labels are continuous). In this work, we develop a novel model for crowdsourcing that can complement standard practices by exploiting people's intuitions about groups and relations between them. We employ a recent machine learning setting, called Ballpark Learning, that can estimate individual labels given only coarse, aggregated signal over groups of data points. To address the important case of continuous labels, we extend the Ballpark setting (which focused on classification) to regression problems. We formulate the problem as a convex optimization problem and propose fast, simple methods with an innate robustness to outliers. We evaluate our methods on real-world datasets, demonstrating how useful constraints about groups can be harnessed from a crowd of non-experts. Our methods can rival supervised models trained on many true labels, and can obtain considerably better results from the crowd than a standard label-collection process (for a lower price). By collecting rough guesses on groups of instances and using machine learning to infer the individual labels, our lightweight framework is able to address core crowdsourcing challenges and train machine learning models in a cost-effective way.

Original languageAmerican English
Title of host publicationWSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining
PublisherAssociation for Computing Machinery, Inc
Pages234-242
Number of pages9
ISBN (Electronic)9781450355810
DOIs
StatePublished - 2 Feb 2018
Event11th ACM International Conference on Web Search and Data Mining, WSDM 2018 - Marina Del Rey, United States
Duration: 5 Feb 20189 Feb 2018

Publication series

NameWSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining
Volume2018-Febuary

Conference

Conference11th ACM International Conference on Web Search and Data Mining, WSDM 2018
Country/TerritoryUnited States
CityMarina Del Rey
Period5/02/189/02/18

Bibliographical note

Publisher Copyright:
© 2018 Association for Computing Machinery.

Fingerprint

Dive into the research topics of 'Ballpark crowdsourcing: The wisdom of rough group comparisons'. Together they form a unique fingerprint.

Cite this