Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models

ArXi:2603.00029v2 Announce Type: replace Large Language Models (LLMs) exhibit highly anisotropic internal representations, often characterized by massive activations, a phenomenon where a small subset of feature dimensions possesses magnitudes significantly larger than the rest. While prior works view these extreme dimensions primarily as artifacts to be managed, we propose a distinct perspective: these dimensions serve as intrinsic interpretable functional units arising from domain specialization.