Type level clustering evaluation: New measures and a POS induction case study

Roi Reichart*, Omri Abend, Ari Rappoport

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

8 Scopus citations

Abstract

Clustering is a central technique in NLP. Consequently, clustering evaluation is of great importance. Many clustering algorithms are evaluated by their success in tagging corpus tokens. In this paper we discuss type level evaluation, which reflects class membership only and is independent of the token statistics of a particular reference corpus. Type level evaluation casts light on the merits of algorithms, and for some applications is a more natural measure of the algorithm's quality. We propose new type level evaluation measures that, contrary to existing measures, are applicable when items are polysemous, the common case in NLP. We demonstrate the benefits of our measures using a detailed case study, POS induction. We experiment with seven leading algorithms, obtaining useful insights and showing that token and type level measures can weakly or even negatively correlate, which underscores the fact that these two approaches reveal different aspects of clustering quality.

Original languageEnglish
Title of host publicationCoNLL 2010 - Fourteenth Conference on Computational Natural Language Learning, Proceedings of the Conference
Pages77-87
Number of pages11
StatePublished - 2010
Event14th Conference on Computational Natural Language Learning, CoNLL 2010 - Uppsala, Sweden
Duration: 15 Jul 201016 Jul 2010

Publication series

NameCoNLL 2010 - Fourteenth Conference on Computational Natural Language Learning, Proceedings of the Conference

Conference

Conference14th Conference on Computational Natural Language Learning, CoNLL 2010
Country/TerritorySweden
CityUppsala
Period15/07/1016/07/10

Fingerprint

Dive into the research topics of 'Type level clustering evaluation: New measures and a POS induction case study'. Together they form a unique fingerprint.

Cite this