Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.
|Original language||American English|
|Title of host publication||5th Conference on Machine Translation, WMT 2020 - Proceedings|
|Editors||Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-Jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Andre Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri|
|Publisher||Association for Computational Linguistics (ACL)|
|Number of pages||8|
|State||Published - 2020|
|Event||5th Conference on Machine Translation, WMT 2020 - Virtual, Online|
Duration: 19 Nov 2020 → 20 Nov 2020
|Name||5th Conference on Machine Translation, WMT 2020 - Proceedings|
|Conference||5th Conference on Machine Translation, WMT 2020|
|Period||19/11/20 → 20/11/20|
Bibliographical noteFunding Information:
This study was supported in parts by the grants 18-24210S of the Czech Science Foundation and 825303 (Bergamot) of the European Union. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (LM2015071).
© 2020 Association for Computational Linguistics