Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

Yossi Arjevani; Yair Carmon; John C. Duchi; Dylan J. Foster; Ayush Sekhari; Karthik Sridharan

Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

Yossi Arjevani^*
, Yair Carmon
, John C. Duchi
, Dylan J. Foster
, Ayush Sekhari
, Karthik Sridharan

^*Corresponding author for this work

Research output: Contribution to journal › Conference article › peer-review

47 Scopus citations

Abstract

We design an algorithm which finds an ε-approximate stationary point (with k∇F(x)k ≤ ε) using O(ε⁻³) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p ≥ 2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ε, γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

Original language	English
Pages (from-to)	242-299
Number of pages	58
Journal	Proceedings of Machine Learning Research
Volume	125
State	Published - 2020
Externally published	Yes
Event	33rd Conference on Learning Theory, COLT 2020 - Virtual, Online, Austria Duration: 9 Jul 2020 → 12 Jul 2020

Bibliographical note

Funding Information:
We thank Blake Woodworth and Nati Srebo for helpful discussions. YA acknowledges partial support from the Sloan Foundation and Samsung Research. JCD acknowledges support from the NSF CAREER award CCF-1553086, ONR YIP N00014-19-2288, Sloan Foundation, NSF HDR 1934578 (Stanford Data Science Collaboratory), and the DAWN Consortium. DF acknowledges the support of TRIPODS award 1740751. KS acknowledges support from NSF CAREER Award 1750575 and a Sloan Research Fellowship.

Publisher Copyright:
© 2020 Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, A. Sekhari & K. Sridharan.

Keywords

Hessian-vector products
Stochastic optimization
non-convex optimization
second-order methods
variance reduction

Access to Document

http://proceedings.mlr.press/v125/arjevani20a.html

Cite this

@article{b2cdd356b07e4a1eae7fe4b523d571d9,

title = "Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations",

abstract = "We design an algorithm which finds an ε-approximate stationary point (with k∇F(x)k ≤ ε) using O(ε−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p ≥ 2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ε, γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.",

keywords = "Hessian-vector products, Stochastic optimization, non-convex optimization, second-order methods, variance reduction",

author = "Yossi Arjevani and Yair Carmon and Duchi, \{John C.\} and Foster, \{Dylan J.\} and Ayush Sekhari and Karthik Sridharan",

note = "Publisher Copyright: {\textcopyright} 2020 Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, A. Sekhari \& K. Sridharan.; 33rd Conference on Learning Theory, COLT 2020 ; Conference date: 09-07-2020 Through 12-07-2020",

year = "2020",

language = "אנגלית",

volume = "125",

pages = "242--299",

journal = "Proceedings of Machine Learning Research",

issn = "2640-3498",

publisher = "ML Research Press",

}

TY - JOUR

T1 - Second-Order Information in Non-Convex Stochastic Optimization

T2 - 33rd Conference on Learning Theory, COLT 2020

AU - Arjevani, Yossi

AU - Carmon, Yair

AU - Duchi, John C.

AU - Foster, Dylan J.

AU - Sekhari, Ayush

AU - Sridharan, Karthik

PY - 2020

Y1 - 2020

N2 - We design an algorithm which finds an ε-approximate stationary point (with k∇F(x)k ≤ ε) using O(ε−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p ≥ 2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ε, γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

AB - We design an algorithm which finds an ε-approximate stationary point (with k∇F(x)k ≤ ε) using O(ε−3) stochastic gradient and Hessian-vector products, matching guarantees that were previously available only under a stronger assumption of access to multiple queries with the same random seed. We prove a lower bound which establishes that this rate is optimal and—surprisingly—that it cannot be improved using stochastic pth order methods for any p ≥ 2, even when the first p derivatives of the objective are Lipschitz. Together, these results characterize the complexity of non-convex stochastic optimization with second-order methods and beyond. Expanding our scope to the oracle complexity of finding (ε, γ)-approximate second-order stationary points, we establish nearly matching upper and lower bounds for stochastic second-order methods. Our lower bounds here are novel even in the noiseless case.

KW - Hessian-vector products

KW - Stochastic optimization

KW - non-convex optimization

KW - second-order methods

KW - variance reduction

UR - https://www.scopus.com/pages/publications/85161288165

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.conferencearticle???

AN - SCOPUS:85161288165

SN - 2640-3498

VL - 125

SP - 242

EP - 299

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

Y2 - 9 July 2020 through 12 July 2020

ER -

Second-Order Information in Non-Convex Stochastic Optimization: Power and Limitations

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this