Gender Coreference and Bias Evaluation at WMT 2020

Tom Kocmi, Tomasz Limisiewicz, Gabriel Stanovsky

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

21 Scopus citations


Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.

Original languageAmerican English
Title of host publication5th Conference on Machine Translation, WMT 2020 - Proceedings
EditorsLoic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-Jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Andre Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri
PublisherAssociation for Computational Linguistics (ACL)
Number of pages8
ISBN (Electronic)9781948087810
StatePublished - 2020
Event5th Conference on Machine Translation, WMT 2020 - Virtual, Online
Duration: 19 Nov 202020 Nov 2020

Publication series

Name5th Conference on Machine Translation, WMT 2020 - Proceedings


Conference5th Conference on Machine Translation, WMT 2020
CityVirtual, Online

Bibliographical note

Publisher Copyright:
© 2020 Association for Computational Linguistics


Dive into the research topics of 'Gender Coreference and Bias Evaluation at WMT 2020'. Together they form a unique fingerprint.

Cite this