AI RESEARCH
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
arXiv CS.LG
•
ArXi:2602.01203v2 Announce Type: replace-cross Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lacking.