Gender Coreference and Bias Evaluation at WMT 2020

Tom Kocmi; Tomasz Limisiewicz; Gabriel Stanovsky

Gender Coreference and Bias Evaluation at WMT 2020

Tom Kocmi, Tomasz Limisiewicz, Gabriel Stanovsky

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

17 Scopus citations

Abstract

Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.

Original language	American English
Title of host publication	5th Conference on Machine Translation, WMT 2020 - Proceedings
Editors	Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-Jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Andre Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri
Publisher	Association for Computational Linguistics (ACL)
Pages	357-364
Number of pages	8
ISBN (Electronic)	9781948087810
State	Published - 2020
Event	5th Conference on Machine Translation, WMT 2020 - Virtual, Online Duration: 19 Nov 2020 → 20 Nov 2020

Publication series

Name	5th Conference on Machine Translation, WMT 2020 - Proceedings

Conference

Conference	5th Conference on Machine Translation, WMT 2020
City	Virtual, Online
Period	19/11/20 → 20/11/20

Bibliographical note

Funding Information:
This study was supported in parts by the grants 18-24210S of the Czech Science Foundation and 825303 (Bergamot) of the European Union. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (LM2015071).

Publisher Copyright:
© 2020 Association for Computational Linguistics

Cite this

Kocmi, T., Limisiewicz, T., & Stanovsky, G. (2020). Gender Coreference and Bias Evaluation at WMT 2020. In L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-Jussa, C. Federmann, M. Fishel, A. Fraser, Y. Graham, P. Guzman, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa, & M. Negri (Eds.), 5th Conference on Machine Translation, WMT 2020 - Proceedings (pp. 357-364). (5th Conference on Machine Translation, WMT 2020 - Proceedings). Association for Computational Linguistics (ACL).

Kocmi, Tom ; Limisiewicz, Tomasz ; Stanovsky, Gabriel. / Gender Coreference and Bias Evaluation at WMT 2020. 5th Conference on Machine Translation, WMT 2020 - Proceedings. editor / Loic Barrault ; Ondrej Bojar ; Fethi Bougares ; Rajen Chatterjee ; Marta R. Costa-Jussa ; Christian Federmann ; Mark Fishel ; Alexander Fraser ; Yvette Graham ; Paco Guzman ; Barry Haddow ; Matthias Huck ; Antonio Jimeno Yepes ; Philipp Koehn ; Andre Martins ; Makoto Morishita ; Christof Monz ; Masaaki Nagata ; Toshiaki Nakazawa ; Matteo Negri. Association for Computational Linguistics (ACL), 2020. pp. 357-364 (5th Conference on Machine Translation, WMT 2020 - Proceedings).

@inproceedings{7907c9700fd04acca175690c741fbf34,

title = "Gender Coreference and Bias Evaluation at WMT 2020",

abstract = "Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.",

author = "Tom Kocmi and Tomasz Limisiewicz and Gabriel Stanovsky",

note = "Funding Information: This study was supported in parts by the grants 18-24210S of the Czech Science Foundation and 825303 (Bergamot) of the European Union. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (LM2015071). Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; 5th Conference on Machine Translation, WMT 2020 ; Conference date: 19-11-2020 Through 20-11-2020",

year = "2020",

language = "American English",

series = "5th Conference on Machine Translation, WMT 2020 - Proceedings",

publisher = "Association for Computational Linguistics (ACL)",

pages = "357--364",

editor = "Loic Barrault and Ondrej Bojar and Fethi Bougares and Rajen Chatterjee and Costa-Jussa, {Marta R.} and Christian Federmann and Mark Fishel and Alexander Fraser and Yvette Graham and Paco Guzman and Barry Haddow and Matthias Huck and Yepes, {Antonio Jimeno} and Philipp Koehn and Andre Martins and Makoto Morishita and Christof Monz and Masaaki Nagata and Toshiaki Nakazawa and Matteo Negri",

booktitle = "5th Conference on Machine Translation, WMT 2020 - Proceedings",

address = "United States",

}

Kocmi, T, Limisiewicz, T & Stanovsky, G 2020, Gender Coreference and Bias Evaluation at WMT 2020. in L Barrault, O Bojar, F Bougares, R Chatterjee, MR Costa-Jussa, C Federmann, M Fishel, A Fraser, Y Graham, P Guzman, B Haddow, M Huck, AJ Yepes, P Koehn, A Martins, M Morishita, C Monz, M Nagata, T Nakazawa & M Negri (eds), 5th Conference on Machine Translation, WMT 2020 - Proceedings. 5th Conference on Machine Translation, WMT 2020 - Proceedings, Association for Computational Linguistics (ACL), pp. 357-364, 5th Conference on Machine Translation, WMT 2020, Virtual, Online, 19/11/20.

Gender Coreference and Bias Evaluation at WMT 2020. / Kocmi, Tom; Limisiewicz, Tomasz; Stanovsky, Gabriel.
5th Conference on Machine Translation, WMT 2020 - Proceedings. ed. / Loic Barrault; Ondrej Bojar; Fethi Bougares; Rajen Chatterjee; Marta R. Costa-Jussa; Christian Federmann; Mark Fishel; Alexander Fraser; Yvette Graham; Paco Guzman; Barry Haddow; Matthias Huck; Antonio Jimeno Yepes; Philipp Koehn; Andre Martins; Makoto Morishita; Christof Monz; Masaaki Nagata; Toshiaki Nakazawa; Matteo Negri. Association for Computational Linguistics (ACL), 2020. p. 357-364 (5th Conference on Machine Translation, WMT 2020 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Gender Coreference and Bias Evaluation at WMT 2020

AU - Kocmi, Tom

AU - Limisiewicz, Tomasz

AU - Stanovsky, Gabriel

N1 - Funding Information: This study was supported in parts by the grants 18-24210S of the Czech Science Foundation and 825303 (Bergamot) of the European Union. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (LM2015071). Publisher Copyright: © 2020 Association for Computational Linguistics

PY - 2020

Y1 - 2020

N2 - Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.

AB - Gender bias in machine translation can manifest when choosing gender inflections based on spurious gender correlations. For example, always translating doctors as men and nurses as women. This can be particularly harmful as models become more popular and deployed within commercial systems. Our work presents the largest evidence for the phenomenon in more than 19 systems submitted to the WMT over four diverse target languages: Czech, German, Polish, and Russian. To achieve this, we use WinoMT, a recent automatic test suite which examines gender coreference and bias when translating from English to languages with grammatical gender. We extend WinoMT to handle two new languages tested in WMT: Polish and Czech. We find that all systems consistently use spurious correlations in the data rather than meaningful contextual information.

UR - http://www.scopus.com/inward/record.url?scp=85108133503&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85108133503

T3 - 5th Conference on Machine Translation, WMT 2020 - Proceedings

SP - 357

EP - 364

BT - 5th Conference on Machine Translation, WMT 2020 - Proceedings

A2 - Barrault, Loic

A2 - Bojar, Ondrej

A2 - Bougares, Fethi

A2 - Chatterjee, Rajen

A2 - Costa-Jussa, Marta R.

A2 - Federmann, Christian

A2 - Fishel, Mark

A2 - Fraser, Alexander

A2 - Graham, Yvette

A2 - Guzman, Paco

A2 - Haddow, Barry

A2 - Huck, Matthias

A2 - Yepes, Antonio Jimeno

A2 - Koehn, Philipp

A2 - Martins, Andre

A2 - Morishita, Makoto

A2 - Monz, Christof

A2 - Nagata, Masaaki

A2 - Nakazawa, Toshiaki

A2 - Negri, Matteo

PB - Association for Computational Linguistics (ACL)

T2 - 5th Conference on Machine Translation, WMT 2020

Y2 - 19 November 2020 through 20 November 2020

ER -

Kocmi T, Limisiewicz T, Stanovsky G. Gender Coreference and Bias Evaluation at WMT 2020. In Barrault L, Bojar O, Bougares F, Chatterjee R, Costa-Jussa MR, Federmann C, Fishel M, Fraser A, Graham Y, Guzman P, Haddow B, Huck M, Yepes AJ, Koehn P, Martins A, Morishita M, Monz C, Nagata M, Nakazawa T, Negri M, editors, 5th Conference on Machine Translation, WMT 2020 - Proceedings. Association for Computational Linguistics (ACL). 2020. p. 357-364. (5th Conference on Machine Translation, WMT 2020 - Proceedings).

Gender Coreference and Bias Evaluation at WMT 2020

Abstract

Publication series

Conference

Bibliographical note

Other files and links

Fingerprint

Cite this