As more computational communication researchers turn to supervised machine learning methods for text classification, we note the challenge in implementing such techniques within an imbalanced dataset. Such issues are critical in our domain, where, in many cases, researchers attempt to identify and study theoretically interesting categories that can be rare in a target corpus. Specifically, imbalanced distributions, with a skewed distribution of texts among the categories, can lead to a lengthy and expensive annotation stage, forcing practitioners to sample and label large numbers of texts to train a classification model. In this paper, we provide an overview of the issue, and describe existing strategies for mitigating such challenges. Noting the pitfalls of previous solutions, we then provide a semi-supervised method–Expert Initiated Latent Space Sampling–that complements researcher domain expertise with a systematic, unsupervised exploration of the latent semantic space to overcome such limitations. Utilizing simulations to systematically evaluate our method and compare it to existing approaches, we show that our procedure offers significant advantages in terms of efficiency and accuracy in many classification tasks.
Bibliographical noteFunding Information:
This work was supported by grants from the Israel Science Foundation (2315/18, 2501/22)Replication Materials can be found online at https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/IEX083 We thank Effi Levi for his input and Tamar David for her assistance in carrying out the simulations within the framework of this project.
© 2023 The Author(s). Published with license by Taylor & Francis Group, LLC.