Constructing free-energy approximations and generalized belief propagation algorithms

Jonathan S. Yedidia*, William T. Freeman, Yair Weiss

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1018 Scopus citations

Abstract

Important inference-problems in statistical physics, computer vision, error-correcting coding theory, and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs. The belief propagation (BP) algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree, but only approximate when the factor graph has cycles. We show that BP fixe d points correspond to the stationary points of the Bethe approximation of the free energy for a factor graph. We explain how to obtain region-based free energy approximations that improve the Bethe approximation, and corresponding generalized belief propagation (GBP) algorithms. We emphasize the conditions a free energy approximation must satisfy in order to be a "valid" or "maxent-normal" approximation. We describe the relationship between four different methods that can be used to generate valid approximations: the "Bethe method," the "junction graph method" the "cluster variation method," and the "region graph method." Finally, we explain how to tell whether a region-based approximation, and its corresponding GBP algorithm, is likely to be accurate, and describe empirical results showing that GBP can significantly outperform BP.

Original languageAmerican English
Pages (from-to)2282-2312
Number of pages31
JournalIEEE Transactions on Information Theory
Volume51
Issue number7
DOIs
StatePublished - Jul 2005

Bibliographical note

Funding Information:
Manuscript received September 2, 2002; revised March 2, 2005. The work of Y. Weiss was supported by the U.S.-Israeli Binational Science Foundation. J. S. Yedidia is with Mitsubishi Electric Research Labs (MERL), Cambridge Research Lab., Cambridge, MA 02139 USA (e-mail: yedidia@merl.com). W. T. Freeman is with the Massachusetts Institute of Technology, Computer Science and Artificial Intelligence Laboratory, Stata Center, D32-476, Cambridge, MA 02139 USA (e-mail: billf@ai.mit.edu). Y. Weiss is with the School of Computer Science and Engineering, The Hebrew University of Jerusalem, 91904 Jerusalem, Israel (e-mail: yweiss@cs.huji.ac.il). Communicated by A. Kavcˇić, Associate Editor for Detection and Estimation. Digital Object Identifier 10.1109/TIT.2005.850085

Keywords

  • Belief propagation (BP)
  • Bethe free energy
  • Cluster variation method
  • Generalized belief propagation (GBP)
  • Kikuchi free energy
  • Message passing
  • Sum-product algorithm

Fingerprint

Dive into the research topics of 'Constructing free-energy approximations and generalized belief propagation algorithms'. Together they form a unique fingerprint.

Cite this