Automatic metric validation for grammatical error correction

Leshem Choshen; Omri Abend

doi:10.18653/v1/p18-1127

Automatic metric validation for grammatical error correction

Leshem Choshen, Omri Abend

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

22 Scopus citations

Abstract

Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M² metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.

Original language	English
Title of host publication	ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
Publisher	Association for Computational Linguistics (ACL)
Pages	1372-1382
Number of pages	11
ISBN (Electronic)	9781948087322
DOIs	https://doi.org/10.18653/v1/p18-1127
State	Published - 2018
Event	56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 - Melbourne, Australia Duration: 15 Jul 2018 → 20 Jul 2018

Publication series

Name	ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
Volume	1

Conference

Conference	56th Annual Meeting of the Association for Computational Linguistics, ACL 2018
Country/Territory	Australia
City	Melbourne
Period	15/07/18 → 20/07/18

Bibliographical note

Publisher Copyright:
© 2018 Association for Computational Linguistics

Access to Document

10.18653/v1/p18-1127

Cite this

Choshen, L., & Abend, O. (2018). Automatic metric validation for grammatical error correction. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers) (pp. 1372-1382). (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers); Vol. 1). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/p18-1127

Choshen, Leshem ; Abend, Omri. / Automatic metric validation for grammatical error correction. ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). Association for Computational Linguistics (ACL), 2018. pp. 1372-1382 (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)).

@inproceedings{4685350097194b08a54f375d697ac3ab,

title = "Automatic metric validation for grammatical error correction",

abstract = "Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.",

author = "Leshem Choshen and Omri Abend",

note = "Publisher Copyright: {\textcopyright} 2018 Association for Computational Linguistics; 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018 ; Conference date: 15-07-2018 Through 20-07-2018",

year = "2018",

doi = "10.18653/v1/p18-1127",

language = "אנגלית",

series = "ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1372--1382",

booktitle = "ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)",

address = "ארצות הברית",

}

Choshen, L & Abend, O 2018, Automatic metric validation for grammatical error correction. in ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), vol. 1, Association for Computational Linguistics (ACL), pp. 1372-1382, 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, 15/07/18. https://doi.org/10.18653/v1/p18-1127

Automatic metric validation for grammatical error correction. / Choshen, Leshem; Abend, Omri.
ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). Association for Computational Linguistics (ACL), 2018. p. 1372-1382 (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers); Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Automatic metric validation for grammatical error correction

AU - Choshen, Leshem

AU - Abend, Omri

PY - 2018

Y1 - 2018

N2 - Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.

AB - Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with MAEGE shed a new light on metric quality, showing for example that the standard M2 metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.

UR - http://www.scopus.com/inward/record.url?scp=85063111386&partnerID=8YFLogxK

U2 - 10.18653/v1/p18-1127

DO - 10.18653/v1/p18-1127

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85063111386

T3 - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)

SP - 1372

EP - 1382

BT - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)

PB - Association for Computational Linguistics (ACL)

T2 - 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018

Y2 - 15 July 2018 through 20 July 2018

ER -

Choshen L, Abend O. Automatic metric validation for grammatical error correction. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). Association for Computational Linguistics (ACL). 2018. p. 1372-1382. (ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)). doi: 10.18653/v1/p18-1127

Automatic metric validation for grammatical error correction

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this