AI RESEARCH
Anatomical Heterogeneity in Transformer Language Models
arXiv CS.LG
•
ArXi:2603.19348v1 Announce Type: new Current transformer language models are trained with uniform computational budgets across all layers, implicitly assuming layer homogeneity. We challenge this assumption through empirical analysis of SmolLM2-135M, a 30-layer, 135M-parameter causal language model, using five diagnostic metrics: weight predictability (R2), ablation degradation, recovery speed, weight manipulation robustness, and structural analysis.