Maximum Likelihood and the Information Bottleneck

Noam Slonim, Yair Weiss

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

11 Scopus citations

Abstract

The information bottleneck (IB) method is an information-theoretic formulation for clustering problems. Given a joint distribution (x, y), this method constructs a new variable that defines partitions over the values of X that are informative about Y. Maximum likelihood (ML) of mixture models is a standard statistical approach to clustering problems. In this paper, we ask: how are the two methods related ? We define a simple mapping between the IB problem and the ML problem for the multinomial mixture model. We show that under this mapping the problems are strongly related. In fact, for uniform input distribution over X or for large sample size, the problems are mathematically equivalent. Specifically, in these cases, every fixed point of the IB-functional defines a fixed point of the (log) likelihood and vice versa. Moreover, the values of the functionals at the fixed points are equal under simple transformations. As a result, in these cases, every algorithm that solves one of the problems, induces a solution for the other.

Original languageAmerican English
Title of host publicationNIPS 2002
Subtitle of host publicationProceedings of the 15th International Conference on Neural Information Processing Systems
EditorsSuzanna Becker, Sebastian Thrun, Klaus Obermayer
PublisherMIT Press Journals
Pages335-342
Number of pages8
ISBN (Electronic)0262025507, 9780262025508
StatePublished - 2002
Event15th International Conference on Neural Information Processing Systems, NIPS 2002 - Vancouver, Canada
Duration: 9 Dec 200214 Dec 2002

Publication series

NameNIPS 2002: Proceedings of the 15th International Conference on Neural Information Processing Systems

Conference

Conference15th International Conference on Neural Information Processing Systems, NIPS 2002
Country/TerritoryCanada
CityVancouver
Period9/12/0214/12/02

Bibliographical note

Publisher Copyright:
© NIPS 2002: Proceedings of the 15th International Conference on Neural Information Processing Systems. All rights reserved.

Fingerprint

Dive into the research topics of 'Maximum Likelihood and the Information Bottleneck'. Together they form a unique fingerprint.

Cite this