AI RESEARCH
Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets
arXiv CS.LG
•
ArXi:2405.17573v3 Announce Type: replace-cross We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth' hyper-parameter $\tilde{L}$. In the infinite depth limit, we study 'representation geodesics' $A_{p}$: continuous paths in representation space (similar to NeuralODEs) from input $p=0$ to output $p=1$ that minimize the parameter norm of the network.