Norm-product belief propagation: Primal-dual message-passing for approximate inference

Tamir Hazan*, Amnon Shashua

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

80 Scopus citations


Inference problems in graphical models can be represented as a constrained optimization of a free-energy function. In this paper, we treat both forms of probabilistic inference, estimating marginal probabilities of the joint distribution and finding the most probable assignment, through a unified message-passing algorithm architecture. In particular we generalize the belief propagation (BP) algorithms of sum-product and max-product and tree-reweighted (TRW) sum and max product algorithms (TRBP) and introduce a new set of convergent algorithms based on "convex-free-energy" and linear-programming (LP) relaxation as a zero-temperature of a convex-free-energy. The main idea of this work arises from taking a general perspective on the existing BP and TRBP algorithms while observing that they all are reductions from the basic optimization formula of f + Σ ihi where the function f is an extended-valued, strictly convex but nonsmooth and the functions hi are extended-valued functions (not necessarily convex). We use tools from convex duality to present the "primal-dual ascent" algorithm which is an extension of the Bregman successive projection scheme and is designed to handle optimization of the general type f + Σihi. We then map the fractional-free-energy variational principle for approximate inference onto the optimization formula above and introduce the "norm-product" message-passing algorithm. Special cases of the norm-product include sum-product and max-product (BP algorithms), TRBP and NMPLP algorithms. When the fractional-free-energy is set to be convex (convex-free-energy) the norm-product is globally convergent for the estimation of marginal probabilities and for approximating the LP-relaxation. We also introduce another branch of the norm-product which arises as the "zero-temperature" of the convex-free-energy which we refer to as the "convex-max-product". The convex-max-product is convergent (unlike max-product) and aims at solving the LP- relaxation.

Original languageAmerican English
Article number5625635
Pages (from-to)6294-6316
Number of pages23
JournalIEEE Transactions on Information Theory
Issue number12
StatePublished - Dec 2010

Bibliographical note

Funding Information:
Manuscript received March 16, 2009; revised June 20, 2010. Date of current version November 19, 2010. This work was supported by the Israeli Science Foundation (ISF) under Grant 519/09. T. Hazan is with Toyota Institute of Technology at Chicago (TTIC), Chicago, IL 60637 USA (e-mail: A. Shashua is with the School of Engineering and Computer Science, The Hebrew University of Jerusalem, Jerusalem 91904, Israel (e-mail: shashua@cs. Communicated by H.-A. Loeliger, Associate Editor for Coding Techniques. Color versions of one or more of the figures in this paper are available online at Digital Object Identifier 10.1109/TIT.2010.2079014


  • Approximate inference
  • Bethe free energy
  • Bregman projection
  • Fenchel duality
  • Markov random fields (MRF)
  • convex free energy
  • dual block ascent
  • graphical models
  • linear programming (LP) relaxation
  • max-product algorithm
  • maximum a posteriori probability (MAP) estimation
  • sum-product algorithm


Dive into the research topics of 'Norm-product belief propagation: Primal-dual message-passing for approximate inference'. Together they form a unique fingerprint.

Cite this