AI RESEARCH
When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models
arXiv CS.LG
•
ArXi:2603.20997v1 Announce Type: new We identify a routing paradox in hybrid recurrent-attention architectures: content-based routing - deciding which tokens deserve expensive attention - requires exactly the pairwise computation that routing is designed to avoid. Through 20+ controlled experiments across three tasks (a synthetic diagnostic, the Zoology MQAR benchmark, and HotpotQA), we map the routing landscape exhaustively. One layer of softmax attention creates a latent ~34-dimensional subspace enabling 98.4% routing precision; zero layers yield 1.2.