TY - GEN

T1 - Maximum likelihood and the information bottleneck

AU - Slonim, Noam

AU - Weiss, Yair

PY - 2003

Y1 - 2003

N2 - The information bottleneck (IB) method is an information-theoretic formulation for clustering problems. Given a joint distributionp(x, y), this method constructs a new variable T that defines partitions over the values of X that are informative about Y. Maximum likelihood (ML) of mixture models is a standard statistical approach to clustering problems. In this paper, we ask: how are the two methods related? We define a simple mapping between the IB problem and the ML problem for the multinomial mixture model. We show that under this mapping the problems are strongly related. In fact, for uniform input distribution over X or for large sample size, the problems are mathematically equivalent. Specifically, in these cases, every fixed point of the IB-functional defines a fixed point of the (log) likelihood and vice versa. Moreover, the values of the functionals at the fixed points are equal under simple transformations. As a result, in these cases, every algorithm that solves one of the problems, induces a solution for the other.

AB - The information bottleneck (IB) method is an information-theoretic formulation for clustering problems. Given a joint distributionp(x, y), this method constructs a new variable T that defines partitions over the values of X that are informative about Y. Maximum likelihood (ML) of mixture models is a standard statistical approach to clustering problems. In this paper, we ask: how are the two methods related? We define a simple mapping between the IB problem and the ML problem for the multinomial mixture model. We show that under this mapping the problems are strongly related. In fact, for uniform input distribution over X or for large sample size, the problems are mathematically equivalent. Specifically, in these cases, every fixed point of the IB-functional defines a fixed point of the (log) likelihood and vice versa. Moreover, the values of the functionals at the fixed points are equal under simple transformations. As a result, in these cases, every algorithm that solves one of the problems, induces a solution for the other.

UR - http://www.scopus.com/inward/record.url?scp=84899009523&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84899009523

SN - 0262025507

SN - 9780262025508

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 15 - Proceedings of the 2002 Conference, NIPS 2002

PB - Neural information processing systems foundation

T2 - 16th Annual Neural Information Processing Systems Conference, NIPS 2002

Y2 - 9 December 2002 through 14 December 2002

ER -