Leveraging Researcher Domain Expertise to Annotate Concepts Within Imbalanced Data

Dror K. Markus*, Guy Mor-Lan, Tamir Sheafer, Shaul R. Shenhav

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


As more computational communication researchers turn to supervised machine learning methods for text classification, we note the challenge in implementing such techniques within an imbalanced dataset. Such issues are critical in our domain, where, in many cases, researchers attempt to identify and study theoretically interesting categories that can be rare in a target corpus. Specifically, imbalanced distributions, with a skewed distribution of texts among the categories, can lead to a lengthy and expensive annotation stage, forcing practitioners to sample and label large numbers of texts to train a classification model. In this paper, we provide an overview of the issue, and describe existing strategies for mitigating such challenges. Noting the pitfalls of previous solutions, we then provide a semi-supervised method–Expert Initiated Latent Space Sampling–that complements researcher domain expertise with a systematic, unsupervised exploration of the latent semantic space to overcome such limitations. Utilizing simulations to systematically evaluate our method and compare it to existing approaches, we show that our procedure offers significant advantages in terms of efficiency and accuracy in many classification tasks.

Original languageAmerican English
Pages (from-to)250-271
Number of pages22
JournalCommunication Methods and Measures
Issue number3
StatePublished - 2023

Bibliographical note

Publisher Copyright:
© 2023 The Author(s). Published with license by Taylor & Francis Group, LLC.


Dive into the research topics of 'Leveraging Researcher Domain Expertise to Annotate Concepts Within Imbalanced Data'. Together they form a unique fingerprint.

Cite this