AI RESEARCH
Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence
arXiv CS.LG
•
ArXi:2604.25930v1 Announce Type: cross We study whether a structured recurrent state can serve as a compact associative backbone for language modeling while still ing exact retrieval. We At small scale, UniMatrix-Core and UniMatrix-ROSA slightly outperform a parameter-matched Transformer on WikiText-2 while using many fewer parameters, reaching 5.084 and 5.083 bits-per-byte versus 5.124.