Maximum likelihood and the information bottleneck

Noam Slonim, Yair Weiss

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

14 Scopus citations


The information bottleneck (IB) method is an information-theoretic formulation for clustering problems. Given a joint distributionp(x, y), this method constructs a new variable T that defines partitions over the values of X that are informative about Y. Maximum likelihood (ML) of mixture models is a standard statistical approach to clustering problems. In this paper, we ask: how are the two methods related? We define a simple mapping between the IB problem and the ML problem for the multinomial mixture model. We show that under this mapping the problems are strongly related. In fact, for uniform input distribution over X or for large sample size, the problems are mathematically equivalent. Specifically, in these cases, every fixed point of the IB-functional defines a fixed point of the (log) likelihood and vice versa. Moreover, the values of the functionals at the fixed points are equal under simple transformations. As a result, in these cases, every algorithm that solves one of the problems, induces a solution for the other.

Original languageAmerican English
Title of host publicationAdvances in Neural Information Processing Systems 15 - Proceedings of the 2002 Conference, NIPS 2002
PublisherNeural information processing systems foundation
ISBN (Print)0262025507, 9780262025508
StatePublished - 2003
Event16th Annual Neural Information Processing Systems Conference, NIPS 2002 - Vancouver, BC, Canada
Duration: 9 Dec 200214 Dec 2002

Publication series

NameAdvances in Neural Information Processing Systems
ISSN (Print)1049-5258


Conference16th Annual Neural Information Processing Systems Conference, NIPS 2002
CityVancouver, BC


Dive into the research topics of 'Maximum likelihood and the information bottleneck'. Together they form a unique fingerprint.

Cite this