AI RESEARCH
Selective Rotary Position Embedding
arXiv CS.LG
•
ArXi:2511.17388v2 Announce Type: replace-cross Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (\textit{RoPE}) encode positions through \textit{fixed-angle} rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we