Efficient clustering of short messages into general domains

Oren Tsur, Adi Littman, Ari Rappoport

Research output: Contribution to conferencePaperpeer-review

43 Scopus citations

Abstract

The ever increasing activity in social networks is mainly manifested by a growing stream of status updating or microblog-ging. The massive stream of updates emphasizes the need for accurate and efficient clustering of short messages on a large scale. Applying traditional clustering techniques is both inaccurate and inefficient due to sparseness. This paper presents an accurate and efficient algorithm for clustering Twitter tweets. We break the clustering task into two distinctive tasks/stages: (1) batch clustering of user annotated data, and (2) online clustering of a stream of tweets. In the first stage we rely on the habit of 'tagging', common in social media streams (e.g. hashtags), thus the algorithm can bootstrap on the tags for clustering of a large pool of hashtagged tweets. The stable clusters achieved in the first stage lend themselves for online clustering of a stream of (mostly) tagless messages. We evaluate our results against gold-standard classification and validate the results by employing multiple clustering evaluation measures (information theoretic, paired, F and greedy). We compare our algorithm to a number of other clustering algorithms and various types of feature sets. Results show that the algorithm presented is both accurate and efficient and can be easily used for large scale clustering of sparse messages as the heavy lifting is achieved on a sublinear number of documents.

Original languageAmerican English
Pages621-630
Number of pages10
StatePublished - 2013
Event7th International AAAI Conference on Weblogs and Social Media, ICWSM 2013 - Cambridge, MA, United States
Duration: 8 Jul 201311 Jul 2013

Conference

Conference7th International AAAI Conference on Weblogs and Social Media, ICWSM 2013
Country/TerritoryUnited States
CityCambridge, MA
Period8/07/1311/07/13

Fingerprint

Dive into the research topics of 'Efficient clustering of short messages into general domains'. Together they form a unique fingerprint.

Cite this