The landscape of NLP research is dominated by large-scale models training on colossal datasets, relying on data quantity rather than quality. As an alternative to this landscape, we propose a method for weighing the relative importance of examples in a dataset based on their Example Training dynamics (ETD; Swayamdipta et al., 2020), a set of metrics computed during training. We propose a new way of computing the ETD of a dataset, and show that they can be used to improve performance in both in-distribution and out-of-distribution testing. We show that ETD can be transferable, i.e., they can be computed once and used for training different models, effectively reducing their computation cost. Finally, we suggest an active learning approach for computing ETD during training rather than as a preprocessing step-an approach that is not as effective, but dramatically reduces the extra computational costs.
|Original language||American English|
|Title of host publication||Findings of the Association for Computational Linguistics, ACL 2023|
|Publisher||Association for Computational Linguistics (ACL)|
|Number of pages||12|
|State||Published - 2023|
|Event||61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 - Toronto, Canada|
Duration: 9 Jul 2023 → 14 Jul 2023
|Name||Proceedings of the Annual Meeting of the Association for Computational Linguistics|
|Conference||61st Annual Meeting of the Association for Computational Linguistics, ACL 2023|
|Period||9/07/23 → 14/07/23|
Bibliographical notePublisher Copyright:
© 2023 Association for Computational Linguistics.