Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures

ArXi:2512.20607v2 Announce Type: replace Neural networks trained with gradient descent often learn solutions of increasing complexity over time, a phenomenon known as simplicity bias. Despite being widely observed across architectures, existing theoretical treatments lack a unifying framework. We present a theoretical framework that explains a simplicity bias arising from saddle-to-saddle learning dynamics for a general class of neural networks, incorporating fully-connected, convolutional, and attention-based architectures.