AI RESEARCH
Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods
arXiv CS.LG
•
ArXi:2312.08531v4 Announce Type: replace In the past several years, the last-iterate convergence of the Stochastic Gradient Descent (SGD) algorithm has triggered people's interest due to its good performance in practice but lack of theoretical understanding. For Lipschitz convex functions, different works have established the optimal $O(\log(1/\delta)\log T/\sqrt{T})$ or $O(\sqrt{\log(1/\delta)/T})$ high-probability convergence rates for the final iterate, where T is the time horizon and \delta is the failure probability.