AI RESEARCH
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
arXiv CS.LG
•
ArXi:2605.05838v1 Announce Type: new Linear Attention (LA) offers a promising paradigm for scaling large language models (LLMs) to long sequences by avoiding the quadratic complexity of self-attention. Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online stochastic gradient descent (SGD), but naive SGD updates suffer from rapid information decay and suboptimal convergence in optimization. While momentum-based optimizers provide a natural remedy, they pose challenges in simultaneously achieving.