AI RESEARCH
OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
arXiv CS.LG
•
ArXi:2605.13473v1 Announce Type: new Linear attention and state-space models offer constant-memory alternatives to softmax attention, but often struggle with in-context associative recall. The Delta Rule mitigates this by writing each token via one step of online gradient descent. However, its step size relies on a single scalar gate that ignores the feature-wise curvature of the inner objective. We propose Online Scaled DeltaNet (OSDN), which augments the scalar gate with a diagonal preconditioner updated online via hypergradient feedback.