The NVI Clustering Evaluation Measure

Roi Reichart; Ari Rappoport

doi:10.3115/1596374.1596401

The NVI Clustering Evaluation Measure

Roi Reichart
, Ari Rappoport

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

33 Scopus citations

Abstract

Clustering is crucial for many NLP tasks and applications. However, evaluating the results of a clustering algorithm is hard. In this paper we focus on the evaluation setting in which a gold standard solution is available. We discuss two existing information theory based measures, V and VI, and show that they are both hard to use when comparing the performance of different algorithms and different datasets. The V measure favors solutions having a large number of clusters, while the range of scores given by VI depends on the size of the dataset. We present a new measure, NVI, which normalizes VI to address the latter problem. We demonstrate the superiority of NVI in a large experiment involving an important NLP application, grammar induction, using real corpus data in English, German and Chinese.

Original language	English
Title of host publication	CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning
Editors	Suzanne Stevenson, Xavier Carreras
Publisher	Association for Computational Linguistics (ACL)
Pages	165-173
Number of pages	9
ISBN (Electronic)	9781932432299
DOIs	https://doi.org/10.3115/1596374.1596401
State	Published - 2009
Event	13th Conference on Computational Natural Language Learning, CoNLL 2009 in conjunction with NAACL HLT - Boulder, United States Duration: 4 Jun 2009 → 5 Jun 2009

Publication series

Name	CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning

Conference

Conference	13th Conference on Computational Natural Language Learning, CoNLL 2009 in conjunction with NAACL HLT
Country/Territory	United States
City	Boulder
Period	4/06/09 → 5/06/09

Bibliographical note

Publisher Copyright:
© 2009 Association for Computational Linguistics.

Access to Document

10.3115/1596374.1596401

Cite this

Reichart, R., & Rappoport, A. (2009). The NVI Clustering Evaluation Measure. In S. Stevenson, & X. Carreras (Eds.), CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning (pp. 165-173). (CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning). Association for Computational Linguistics (ACL). https://doi.org/10.3115/1596374.1596401

Reichart, Roi ; Rappoport, Ari. / The NVI Clustering Evaluation Measure. CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning. editor / Suzanne Stevenson ; Xavier Carreras. Association for Computational Linguistics (ACL), 2009. pp. 165-173 (CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning).

@inproceedings{4382559d52074689a1e0b808f90e7b85,

title = "The NVI Clustering Evaluation Measure",

abstract = "Clustering is crucial for many NLP tasks and applications. However, evaluating the results of a clustering algorithm is hard. In this paper we focus on the evaluation setting in which a gold standard solution is available. We discuss two existing information theory based measures, V and VI, and show that they are both hard to use when comparing the performance of different algorithms and different datasets. The V measure favors solutions having a large number of clusters, while the range of scores given by VI depends on the size of the dataset. We present a new measure, NVI, which normalizes VI to address the latter problem. We demonstrate the superiority of NVI in a large experiment involving an important NLP application, grammar induction, using real corpus data in English, German and Chinese.",

author = "Roi Reichart and Ari Rappoport",

note = "Publisher Copyright: {\textcopyright} 2009 Association for Computational Linguistics.; 13th Conference on Computational Natural Language Learning, CoNLL 2009 in conjunction with NAACL HLT ; Conference date: 04-06-2009 Through 05-06-2009",

year = "2009",

doi = "10.3115/1596374.1596401",

language = "אנגלית",

series = "CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning",

publisher = "Association for Computational Linguistics (ACL)",

pages = "165--173",

editor = "Suzanne Stevenson and Xavier Carreras",

booktitle = "CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning",

address = "ארצות הברית",

}

Reichart, R & Rappoport, A 2009, The NVI Clustering Evaluation Measure. in S Stevenson & X Carreras (eds), CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning. CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning, Association for Computational Linguistics (ACL), pp. 165-173, 13th Conference on Computational Natural Language Learning, CoNLL 2009 in conjunction with NAACL HLT, Boulder, United States, 4/06/09. https://doi.org/10.3115/1596374.1596401

The NVI Clustering Evaluation Measure. / Reichart, Roi; Rappoport, Ari.
CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning. ed. / Suzanne Stevenson; Xavier Carreras. Association for Computational Linguistics (ACL), 2009. p. 165-173 (CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The NVI Clustering Evaluation Measure

AU - Reichart, Roi

AU - Rappoport, Ari

PY - 2009

Y1 - 2009

N2 - Clustering is crucial for many NLP tasks and applications. However, evaluating the results of a clustering algorithm is hard. In this paper we focus on the evaluation setting in which a gold standard solution is available. We discuss two existing information theory based measures, V and VI, and show that they are both hard to use when comparing the performance of different algorithms and different datasets. The V measure favors solutions having a large number of clusters, while the range of scores given by VI depends on the size of the dataset. We present a new measure, NVI, which normalizes VI to address the latter problem. We demonstrate the superiority of NVI in a large experiment involving an important NLP application, grammar induction, using real corpus data in English, German and Chinese.

AB - Clustering is crucial for many NLP tasks and applications. However, evaluating the results of a clustering algorithm is hard. In this paper we focus on the evaluation setting in which a gold standard solution is available. We discuss two existing information theory based measures, V and VI, and show that they are both hard to use when comparing the performance of different algorithms and different datasets. The V measure favors solutions having a large number of clusters, while the range of scores given by VI depends on the size of the dataset. We present a new measure, NVI, which normalizes VI to address the latter problem. We demonstrate the superiority of NVI in a large experiment involving an important NLP application, grammar induction, using real corpus data in English, German and Chinese.

UR - https://www.scopus.com/pages/publications/84858319308

U2 - 10.3115/1596374.1596401

DO - 10.3115/1596374.1596401

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:84858319308

T3 - CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning

SP - 165

EP - 173

BT - CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning

A2 - Stevenson, Suzanne

A2 - Carreras, Xavier

PB - Association for Computational Linguistics (ACL)

T2 - 13th Conference on Computational Natural Language Learning, CoNLL 2009 in conjunction with NAACL HLT

Y2 - 4 June 2009 through 5 June 2009

ER -

Reichart R, Rappoport A. The NVI Clustering Evaluation Measure. In Stevenson S, Carreras X, editors, CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning. Association for Computational Linguistics (ACL). 2009. p. 165-173. (CoNLL 2009 - Proceedings of the 13th Conference on Computational Natural Language Learning). doi: 10.3115/1596374.1596401

The NVI Clustering Evaluation Measure

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this