Proving the Lottery Ticket Hypothesis: Pruning is All You Need

Eran Malach; Gilad Yehudai; Shai Shalev-Shwartz; Ohad Shamir

Proving the Lottery Ticket Hypothesis: Pruning is All You Need

Eran Malach^*
, Gilad Yehudai^*
, Shai Shalev-Shwartz
, Ohad Shamir

^*Corresponding author for this work

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Contribution to journal › Conference article › peer-review

54 Scopus citations

Abstract

The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the per- formance of the original network. We prove an even stronger hypothesis (as was also con- jectured in Ramanujan et al., 2019), showing that for every bounded distribution and every tar- get network with bounded weights, a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.

Original language	English
Journal	Proceedings of Machine Learning Research
Volume	119
State	Published - 2020
Event	37th International Conference on Machine Learning, ICML 2020 - Virtual, Online Duration: 13 Jul 2020 → 18 Jul 2020

Bibliographical note

Publisher Copyright:
© 2020 by the author(s).

Cite this

@article{5bc068f46cb2442caa5ba7b2cd11b7d1,

title = "Proving the Lottery Ticket Hypothesis: Pruning is All You Need",

abstract = "The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the per- formance of the original network. We prove an even stronger hypothesis (as was also con- jectured in Ramanujan et al., 2019), showing that for every bounded distribution and every tar- get network with bounded weights, a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.",

author = "Eran Malach and Gilad Yehudai and Shai Shalev-Shwartz and Ohad Shamir",

note = "Publisher Copyright: {\textcopyright} 2020 by the author(s).; 37th International Conference on Machine Learning, ICML 2020 ; Conference date: 13-07-2020 Through 18-07-2020",

year = "2020",

language = "אנגלית",

volume = "119",

journal = "Proceedings of Machine Learning Research",

issn = "2640-3498",

publisher = "ML Research Press",

}

TY - JOUR

T1 - Proving the Lottery Ticket Hypothesis

T2 - 37th International Conference on Machine Learning, ICML 2020

AU - Malach, Eran

AU - Yehudai, Gilad

AU - Shalev-Shwartz, Shai

AU - Shamir, Ohad

PY - 2020

Y1 - 2020

N2 - The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the per- formance of the original network. We prove an even stronger hypothesis (as was also con- jectured in Ramanujan et al., 2019), showing that for every bounded distribution and every tar- get network with bounded weights, a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.

AB - The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the per- formance of the original network. We prove an even stronger hypothesis (as was also con- jectured in Ramanujan et al., 2019), showing that for every bounded distribution and every tar- get network with bounded weights, a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.

UR - https://www.scopus.com/pages/publications/105022247834

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.conferencearticle???

AN - SCOPUS:105022247834

SN - 2640-3498

VL - 119

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

Y2 - 13 July 2020 through 18 July 2020

ER -

Proving the Lottery Ticket Hypothesis: Pruning is All You Need

Abstract

Bibliographical note

Other files and links

Fingerprint

Cite this