Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts

ArXi:2603.07323v1 Announce Type: new Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions and that neural networks exhibit simplicity bias, but neither explains the timescale of the transition from shortcut features to structured representations.