Critical Slowing down near Topological Transitions in Rate-Distortion Problems

Shlomi Agrnon; Etam Benger; Or Ordentlich; Naftali Tishby

doi:10.1109/ISIT45174.2021.9517956

Critical Slowing down near Topological Transitions in Rate-Distortion Problems

Shlomi Agrnon, Etam Benger, Or Ordentlich, Naftali Tishby

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the ratedistortion and information bottleneck settings. We argue that they suffer from critical slowing down - a diverging number of iterations for convergence - near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems.

Original language	English
Title of host publication	2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2625-2630
Number of pages	6
ISBN (Electronic)	9781538682098
DOIs	https://doi.org/10.1109/ISIT45174.2021.9517956
State	Published - 12 Jul 2021
Event	2021 IEEE International Symposium on Information Theory, ISIT 2021 - Virtual, Melbourne, Australia Duration: 12 Jul 2021 → 20 Jul 2021

Publication series

Name	IEEE International Symposium on Information Theory - Proceedings
Volume	2021-July
ISSN (Print)	2157-8095

Conference

Conference	2021 IEEE International Symposium on Information Theory, ISIT 2021
Country/Territory	Australia
City	Virtual, Melbourne
Period	12/07/21 → 20/07/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Access to Document

10.1109/ISIT45174.2021.9517956

Cite this

Agrnon, S., Benger, E., Ordentlich, O., & Tishby, N. (2021). Critical Slowing down near Topological Transitions in Rate-Distortion Problems. In 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings (pp. 2625-2630). (IEEE International Symposium on Information Theory - Proceedings; Vol. 2021-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT45174.2021.9517956

@inproceedings{89907af9c00d43998e0a180ab8b218b7,

title = "Critical Slowing down near Topological Transitions in Rate-Distortion Problems",

abstract = "In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the ratedistortion and information bottleneck settings. We argue that they suffer from critical slowing down - a diverging number of iterations for convergence - near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems.",

author = "Shlomi Agrnon and Etam Benger and Or Ordentlich and Naftali Tishby",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE International Symposium on Information Theory, ISIT 2021 ; Conference date: 12-07-2021 Through 20-07-2021",

year = "2021",

month = jul,

day = "12",

doi = "10.1109/ISIT45174.2021.9517956",

language = "אנגלית",

series = "IEEE International Symposium on Information Theory - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2625--2630",

booktitle = "2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings",

address = "ארצות הברית",

}

Agrnon, S, Benger, E, Ordentlich, O & Tishby, N 2021, Critical Slowing down near Topological Transitions in Rate-Distortion Problems. in 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings. IEEE International Symposium on Information Theory - Proceedings, vol. 2021-July, Institute of Electrical and Electronics Engineers Inc., pp. 2625-2630, 2021 IEEE International Symposium on Information Theory, ISIT 2021, Virtual, Melbourne, Australia, 12/07/21. https://doi.org/10.1109/ISIT45174.2021.9517956

Critical Slowing down near Topological Transitions in Rate-Distortion Problems. / Agrnon, Shlomi; Benger, Etam; Ordentlich, Or et al.
2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2021. p. 2625-2630 (IEEE International Symposium on Information Theory - Proceedings; Vol. 2021-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Critical Slowing down near Topological Transitions in Rate-Distortion Problems

AU - Agrnon, Shlomi

AU - Benger, Etam

AU - Ordentlich, Or

AU - Tishby, Naftali

PY - 2021/7/12

Y1 - 2021/7/12

N2 - In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the ratedistortion and information bottleneck settings. We argue that they suffer from critical slowing down - a diverging number of iterations for convergence - near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems.

AB - In rate-distortion (RD) problems one seeks reduced representations of a source that meet a target distortion constraint. Such optimal representations undergo topological transitions at some critical rate values, when their cardinality or dimensionality change. We study the convergence time of the Arimoto-Blahut alternating projection algorithms, used to solve such problems, near those critical points, both for the ratedistortion and information bottleneck settings. We argue that they suffer from critical slowing down - a diverging number of iterations for convergence - near the critical points. This phenomenon can have theoretical and practical implications for both machine learning and data compression problems.

UR - http://www.scopus.com/inward/record.url?scp=85115053375&partnerID=8YFLogxK

U2 - 10.1109/ISIT45174.2021.9517956

DO - 10.1109/ISIT45174.2021.9517956

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85115053375

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 2625

EP - 2630

BT - 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2021 IEEE International Symposium on Information Theory, ISIT 2021

Y2 - 12 July 2021 through 20 July 2021

ER -

Agrnon S, Benger E, Ordentlich O, Tishby N. Critical Slowing down near Topological Transitions in Rate-Distortion Problems. In 2021 IEEE International Symposium on Information Theory, ISIT 2021 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2021. p. 2625-2630. (IEEE International Symposium on Information Theory - Proceedings). doi: 10.1109/ISIT45174.2021.9517956

Critical Slowing down near Topological Transitions in Rate-Distortion Problems

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this