AI RESEARCH
Linear Predictability of Attention Heads in Large Language Models
arXiv CS.LG
•
ArXi:2603.13314v1 Announce Type: new Large language model (LLM) inference is increasingly bottlenecked by the Key-Value (KV) cache, yet the fine-grained structure of attention-head activations remains poorly understood. We show that pretrained Transformers exhibit a pervasive inter-head linear structure: for a given token, the Query, Key, and Value (QKV) vectors of an attention head can often be reconstructed as a linear combination of a small number of peer heads, typically within the same layer.