On graduated optimization for stochastic non-convex problems

The graduated optimization approach, also known as the continuation method, is a popular heuristic to solving non-convcx problems that has received renewed interest over the last decade. Despite being popular, very little is known in terms of its theoretical convergence analysis. In this paper we describe a new first-order algorithm based on graduated optimization and analyze its performance. We characterize a family of non-convex functions for which this algorithm provably converges to a global optimum. In particular, we prove that the algorithm converges to an ϵ-approximate solution within O( 1/ϵ⁴) gradient-based steps. We extend our algorithm and analysis to the setting of stochastic non-convex optimization with noisy gradient feedback, attaining the same convergence rate. Additionally, we discuss the setting of "zero- order optimization", and devise a variant of our algorithm which converges at rate of O(d²/ϵ⁴).