AI RESEARCH
Convergent Evolution: How Different Language Models Learn Similar Number Representations
arXiv CS.LG
•
ArXi:2604.20817v1 Announce Type: cross Language models trained on natural text learn to represent numbers using periodic features with dominant periods at $T=2, 5, 10$. In this paper, we identify a two-tiered hierarchy of these features: while Transformers, Linear RNNs, LSTMs, and classical word embeddings trained in different ways all learn features that have period-$T$ spikes in the Fourier domain, only some learn geometrically separable features that can be used to linearly classify a number mod-$T.