Lower bounds for non-convex stochastic optimization

Yossi Arjevani; Yair Carmon; John C. Duchi; Dylan J. Foster; Nathan Srebro; Blake Woodworth

doi:10.1007/s10107-022-01822-7

Lower bounds for non-convex stochastic optimization

Yossi Arjevani, Yair Carmon^*, John C. Duchi, Dylan J. Foster, Nathan Srebro, Blake Woodworth

^*Corresponding author for this work

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Contribution to journal › Article › peer-review

2 Scopus citations

Abstract

We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ^{- 4} queries to find an ϵ-stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ^{- 3} queries, establishing the optimality of recently proposed variance reduction techniques.

Original language	American English
Pages (from-to)	165-214
Number of pages	50
Journal	Mathematical Programming
Volume	199
Issue number	1-2
DOIs	https://doi.org/10.1007/s10107-022-01822-7
State	Published - 2023

Bibliographical note

Funding Information:
Part of this work was completed while the authors were visiting the Simons Institute for the Foundations of Deep Learning program. We thank Ayush Sekhari, Ohad Shamir, Aaron Sidford and Karthik Sridharan for several helpful discussions. YC was supported by the Stanford Graduate Fellowship. JCD acknowledges support from NSF CAREER award 1553086, the Sloan Foundation, and ONR-YIP N00014-19-1-2288. DF was supported by NSF TRIPODS award #1740751. BW was supported by the Google PhD Fellowship program. Division of Computing and Communication Foundations (Grant Number 1553086

Publisher Copyright:
© 2022, Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society.

Access to Document

10.1007/s10107-022-01822-7

Cite this

@article{53ea803bb5474b99bd8ce4b76f309d53,

title = "Lower bounds for non-convex stochastic optimization",

abstract = "We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ- 4 queries to find an ϵ-stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ- 3 queries, establishing the optimality of recently proposed variance reduction techniques.",

author = "Yossi Arjevani and Yair Carmon and Duchi, {John C.} and Foster, {Dylan J.} and Nathan Srebro and Blake Woodworth",

note = "Funding Information: Part of this work was completed while the authors were visiting the Simons Institute for the Foundations of Deep Learning program. We thank Ayush Sekhari, Ohad Shamir, Aaron Sidford and Karthik Sridharan for several helpful discussions. YC was supported by the Stanford Graduate Fellowship. JCD acknowledges support from NSF CAREER award 1553086, the Sloan Foundation, and ONR-YIP N00014-19-1-2288. DF was supported by NSF TRIPODS award #1740751. BW was supported by the Google PhD Fellowship program. Division of Computing and Communication Foundations (Grant Number 1553086 Publisher Copyright: {\textcopyright} 2022, Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society.",

year = "2023",

doi = "10.1007/s10107-022-01822-7",

language = "American English",

volume = "199",

pages = "165--214",

journal = "Mathematical Programming",

issn = "0025-5610",

publisher = "Springer-Verlag GmbH and Co. KG",

number = "1-2",

}

TY - JOUR

T1 - Lower bounds for non-convex stochastic optimization

AU - Arjevani, Yossi

AU - Carmon, Yair

AU - Duchi, John C.

AU - Foster, Dylan J.

AU - Srebro, Nathan

AU - Woodworth, Blake

N1 - Funding Information: Part of this work was completed while the authors were visiting the Simons Institute for the Foundations of Deep Learning program. We thank Ayush Sekhari, Ohad Shamir, Aaron Sidford and Karthik Sridharan for several helpful discussions. YC was supported by the Stanford Graduate Fellowship. JCD acknowledges support from NSF CAREER award 1553086, the Sloan Foundation, and ONR-YIP N00014-19-1-2288. DF was supported by NSF TRIPODS award #1740751. BW was supported by the Google PhD Fellowship program. Division of Computing and Communication Foundations (Grant Number 1553086 Publisher Copyright: © 2022, Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society.

PY - 2023

Y1 - 2023

N2 - We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ- 4 queries to find an ϵ-stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ- 3 queries, establishing the optimality of recently proposed variance reduction techniques.

AB - We lower bound the complexity of finding ϵ-stationary points (with gradient norm at most ϵ) using stochastic first-order methods. In a well-studied model where algorithms access smooth, potentially non-convex functions through queries to an unbiased stochastic gradient oracle with bounded variance, we prove that (in the worst case) any algorithm requires at least ϵ- 4 queries to find an ϵ-stationary point. The lower bound is tight, and establishes that stochastic gradient descent is minimax optimal in this model. In a more restrictive model where the noisy gradient estimates satisfy a mean-squared smoothness property, we prove a lower bound of ϵ- 3 queries, establishing the optimality of recently proposed variance reduction techniques.

UR - http://www.scopus.com/inward/record.url?scp=85131593123&partnerID=8YFLogxK

U2 - 10.1007/s10107-022-01822-7

DO - 10.1007/s10107-022-01822-7

M3 - Article

AN - SCOPUS:85131593123

SN - 0025-5610

VL - 199

SP - 165

EP - 214

JO - Mathematical Programming

JF - Mathematical Programming

IS - 1-2

ER -

Lower bounds for non-convex stochastic optimization

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this