AI RESEARCH
Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent
arXiv CS.LG
•
ArXi:2110.11442v4 Announce Type: replace-cross We aim to make stochastic gradient descent (SGD) adaptive to (i) the noise $\sigma^2$ in the stochastic gradients and (ii) problem-dependent constants. When minimizing smooth, strongly-convex functions with condition number $\kappa$, we prove that $T$ iterations of SGD with exponentially decreasing step-sizes and knowledge of the smoothness can achieve an $\tilde{O} \left(\exp \left( \frac{-T}{\kappa} \right) + \frac{\sigma^2}{T} \right)$ rate, without knowing $\sigma^2.