Most Neural Networks Are Almost Learnable

Amit Daniely; Nathan Srebro; Gal Vardi

Most Neural Networks Are Almost Learnable

Amit Daniely, Nathan Srebro, Gal Vardi

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We present a PTAS for learning random constant-depth networks. We show that for any fixed ϵ > 0 and depth i, there is a poly-time algorithm that for any distribution on √d· S^d−1 learns random Xavier networks of depth i, up to an additive error of ϵ. The algorithm runs in time and sample complexity of (d̄)^poly(ϵ−1), where d̄is the size of the network. For some cases of sigmoid and ReLU-like activations the bound can be improved to (d̄)^{polylog(ϵ−1)}, resulting in a quasi-poly-time algorithm for learning constant depth random networks.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023
Editors	A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, S. Levine
Publisher	Neural information processing systems foundation
ISBN (Electronic)	9781713899921
State	Published - 2023
Event	37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States Duration: 10 Dec 2023 → 16 Dec 2023 Conference number: 37

Publication series

Name	Advances in Neural Information Processing Systems
Volume	36
ISSN (Print)	1049-5258

Conference

Conference	37th Conference on Neural Information Processing Systems, NeurIPS 2023
Country/Territory	United States
City	New Orleans
Period	10/12/23 → 16/12/23

Bibliographical note

Publisher Copyright:
© 2023 Neural information processing systems foundation. All rights reserved.

Cite this

Daniely, A., Srebro, N., & Vardi, G. (2023). Most Neural Networks Are Almost Learnable. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023 (Advances in Neural Information Processing Systems; Vol. 36). Neural information processing systems foundation.

Daniely, Amit ; Srebro, Nathan ; Vardi, Gal. / Most Neural Networks Are Almost Learnable. Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. editor / A. Oh ; T. Neumann ; A. Globerson ; K. Saenko ; M. Hardt ; S. Levine. Neural information processing systems foundation, 2023. (Advances in Neural Information Processing Systems).

@inproceedings{9896d2880bc744409385f5f0073d1888,

title = "Most Neural Networks Are Almost Learnable",

abstract = "We present a PTAS for learning random constant-depth networks. We show that for any fixed ϵ > 0 and depth i, there is a poly-time algorithm that for any distribution on √d· Sd−1 learns random Xavier networks of depth i, up to an additive error of ϵ. The algorithm runs in time and sample complexity of ({\=d})poly(ϵ−1), where {\=d}is the size of the network. For some cases of sigmoid and ReLU-like activations the bound can be improved to ({\=d})polylog(ϵ−1), resulting in a quasi-poly-time algorithm for learning constant depth random networks.",

author = "Amit Daniely and Nathan Srebro and Gal Vardi",

note = "Publisher Copyright: {\textcopyright} 2023 Neural information processing systems foundation. All rights reserved.; 37th Conference on Neural Information Processing Systems, NeurIPS 2023 ; Conference date: 10-12-2023 Through 16-12-2023",

year = "2023",

language = "אנגלית",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

editor = "A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine",

booktitle = "Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023",

}

Daniely, A, Srebro, N & Vardi, G 2023, Most Neural Networks Are Almost Learnable. in A Oh, T Neumann, A Globerson, K Saenko, M Hardt & S Levine (eds), Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. Advances in Neural Information Processing Systems, vol. 36, Neural information processing systems foundation, 37th Conference on Neural Information Processing Systems, NeurIPS 2023, New Orleans, Louisiana, United States, 10/12/23.

Most Neural Networks Are Almost Learnable. / Daniely, Amit; Srebro, Nathan; Vardi, Gal.
Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. ed. / A. Oh; T. Neumann; A. Globerson; K. Saenko; M. Hardt; S. Levine. Neural information processing systems foundation, 2023. (Advances in Neural Information Processing Systems; Vol. 36).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Most Neural Networks Are Almost Learnable

AU - Daniely, Amit

AU - Srebro, Nathan

AU - Vardi, Gal

N1 - Conference code: 37

PY - 2023

Y1 - 2023

N2 - We present a PTAS for learning random constant-depth networks. We show that for any fixed ϵ > 0 and depth i, there is a poly-time algorithm that for any distribution on √d· Sd−1 learns random Xavier networks of depth i, up to an additive error of ϵ. The algorithm runs in time and sample complexity of (d̄)poly(ϵ−1), where d̄is the size of the network. For some cases of sigmoid and ReLU-like activations the bound can be improved to (d̄)polylog(ϵ−1), resulting in a quasi-poly-time algorithm for learning constant depth random networks.

AB - We present a PTAS for learning random constant-depth networks. We show that for any fixed ϵ > 0 and depth i, there is a poly-time algorithm that for any distribution on √d· Sd−1 learns random Xavier networks of depth i, up to an additive error of ϵ. The algorithm runs in time and sample complexity of (d̄)poly(ϵ−1), where d̄is the size of the network. For some cases of sigmoid and ReLU-like activations the bound can be improved to (d̄)polylog(ϵ−1), resulting in a quasi-poly-time algorithm for learning constant depth random networks.

UR - http://www.scopus.com/inward/record.url?scp=85191175143&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85191175143

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

A2 - Oh, A.

A2 - Neumann, T.

A2 - Globerson, A.

A2 - Saenko, K.

A2 - Hardt, M.

A2 - Levine, S.

PB - Neural information processing systems foundation

T2 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

Y2 - 10 December 2023 through 16 December 2023

ER -

Daniely A, Srebro N, Vardi G. Most Neural Networks Are Almost Learnable. In Oh A, Neumann T, Globerson A, Saenko K, Hardt M, Levine S, editors, Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. Neural information processing systems foundation. 2023. (Advances in Neural Information Processing Systems).

Most Neural Networks Are Almost Learnable

Abstract

Publication series

Conference

Bibliographical note

Other files and links

Fingerprint

Cite this