AI RESEARCH
Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis
arXiv CS.LG
•
ArXi:2601.11789v2 Announce Type: replace This paper explores the suspicious alignment phenomenon in stochastic gradient descent (SGD) under ill-conditioned optimization, where the Hessian spectrum splits into dominant and bulk subspaces. This phenomenon describes the behavior of gradient alignment in SGD updates. Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease. Subsequently, it enters a rising phase and eventually stabilizes in a high-alignment phase.