AI RESEARCH
SAGA: Selective Adaptive Gating for Efficient and Expressive Linear Attention
arXiv CS.CV
•
ArXi:2509.12817v2 Announce Type: replace While Transformer architecture excel at modeling long-range dependencies contributing to its widespread adoption in vision tasks the quadratic complexity of softmax-based attention mechanisms imposes a major bottleneck, particularly when processing high-resolution images. Linear attention presents a promising alternative by reformulating the attention computation from $(QK)V$ to $Q(KV)$, thereby reducing the complexity from $\mathcal{O}(N^2)$ to $\mathcal{O}(N)$ while preserving the global receptive field.