AI RESEARCH
SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds
arXiv CS.LG
•
ArXi:2603.16535v1 Announce Type: new Transformers owe much of their empirical success in natural language processing to the self-attention blocks. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of interaction energy functionals on probability density spaces equipped with Wasserstein-$2$-type metrics. We extend this viewpoint by