A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates

Yossi Arjevani, Ohad Shamir, Nathan Srebro

Research output: Contribution to journalConference articlepeer-review

25 Scopus citations

Abstract

We establish matching upper and lower complexity bounds for gradient descent and stochastic gradient descent on quadratic functions, when the gradients are delayed and reflect iterates from τ rounds ago. First, we show that without stochastic noise, delays strongly affect the attainable optimization error: In fact, the error can be as bad as non-delayed gradient descent ran on only 1/τ of the gradients. In sharp contrast, we quantify how stochastic noise makes the effect of delays negligible, improving on previous work which only showed this phenomenon asymptotically or for much smaller delays. Also, in the context of distributed optimization, the results indicate that the performance of gradient descent with delays is competitive with synchronous approaches such as mini-batching. Our results are based on a novel technique for analyzing convergence of optimization algorithms using generating functions.

Original languageAmerican English
Pages (from-to)111-132
Number of pages22
JournalProceedings of Machine Learning Research
Volume117
StatePublished - 2020
Externally publishedYes
Event31st International Conference on Algorithmic Learning Theory, ALT 2020 - San Diego, United States
Duration: 8 Feb 202011 Feb 2020

Bibliographical note

Funding Information:
This research is partially supported by an NSF/BSF grant no. 2016741.

Publisher Copyright:
© 2020 Y. Arjevani, O. Shamir & N. Srebro.

Keywords

  • asynchronous
  • delayed
  • lower bounds
  • optimization
  • stochastic gradient descent
  • upper bounds

Fingerprint

Dive into the research topics of 'A Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates'. Together they form a unique fingerprint.

Cite this