Leveraging Researcher Domain Expertise to Annotate Concepts Within Imbalanced Data

Dror K. Markus*, Guy Mor-Lan, Tamir Sheafer, Shaul R. Shenhav

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


As more computational communication researchers turn to supervised machine learning methods for text classification, we note the challenge in implementing such techniques within an imbalanced dataset. Such issues are critical in our domain, where, in many cases, researchers attempt to identify and study theoretically interesting categories that can be rare in a target corpus. Specifically, imbalanced distributions, with a skewed distribution of texts among the categories, can lead to a lengthy and expensive annotation stage, forcing practitioners to sample and label large numbers of texts to train a classification model. In this paper, we provide an overview of the issue, and describe existing strategies for mitigating such challenges. Noting the pitfalls of previous solutions, we then provide a semi-supervised method–Expert Initiated Latent Space Sampling–that complements researcher domain expertise with a systematic, unsupervised exploration of the latent semantic space to overcome such limitations. Utilizing simulations to systematically evaluate our method and compare it to existing approaches, we show that our procedure offers significant advantages in terms of efficiency and accuracy in many classification tasks.

Original languageAmerican English
JournalCommunication Methods and Measures
StateAccepted/In press - 2023

Bibliographical note

Funding Information:
This work was supported by grants from the Israel Science Foundation (2315/18, 2501/22)Replication Materials can be found online at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/IEX083 We thank Effi Levi for his input and Tamar David for her assistance in carrying out the simulations within the framework of this project.

Publisher Copyright:
© 2023 The Author(s). Published with license by Taylor & Francis Group, LLC.


Dive into the research topics of 'Leveraging Researcher Domain Expertise to Annotate Concepts Within Imbalanced Data'. Together they form a unique fingerprint.

Cite this