Design Principles for Sequence Models via Coefficient Dynamics

ArXi:2510.09389v2 Announce Type: replace-cross Deep sequence models, ranging from Transformers and State Space Models (SSMs) to recent approaches such as gated linear RNNs, fundamentally compute outputs as linear combinations of past value vectors. To draw insights and systematically compare such architectures, we develop a unified framework that makes this output operation explicit, by casting the linear combination coefficients as the outputs of autonomous linear dynamical systems driven by impulse inputs.