AI RESEARCH
As Language Models Scale, Low-order Linear Depth Dynamics Emerge
arXiv CS.LG
•
ArXi:2603.12541v1 Announce Type: new Large language models are often viewed as high-dimensional nonlinear systems and treated as black boxes. Here, we show that transformer depth dynamics admit accurate low-order linear surrogates within context. Across tasks including toxicity, irony, hate speech and sentiment, a 32-dimensional linear surrogate reproduces the layerwise sensitivity profile of GPT-2-large with near-perfect agreement, capturing how the final output shifts under additive injections at each layer.