Multivariate information bottleneck

Noam Slonim*, Nir Friedman, Naftali Tishby

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

66 Scopus citations

Abstract

The information bottleneck (IB) method is an unsupervised model independent data organization technique. Given a joint distribution, p(X, Y), this method constructs a new variable, T, that extracts partitions, or clusters, over the values of X that are informative about Y. Algorithms that are motivated by the IB method have already been applied to text classification, gene expression, neural code, and spectral analysis. Here, we introduce a general principled framework for multivariate extensions of the IB method. This allows us to consider multiple systems of data partitions that are interrelated. Our approach utilizes Bayesian networks for specifying the systems of clusters and which information terms should be maintained. We show that this construction provides insights about bottleneck variations and enables us to characterize the solutions of these variations. We also present four different algorithmic approaches that allow us to construct solutions in practice and apply them to several real-world problems.

Original languageAmerican English
Pages (from-to)1739-1789
Number of pages51
JournalNeural Computation
Volume18
Issue number8
DOIs
StatePublished - 2006

Fingerprint

Dive into the research topics of 'Multivariate information bottleneck'. Together they form a unique fingerprint.

Cite this