AI RESEARCH
Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition
arXiv CS.LG
•
ArXi:2605.10466v1 Announce Type: new Large language models (LLMs) exhibit two striking and ostensibly unrelated behaviours: in-context learning (ICL) and repetitive generation. In both, the model behaves as though it had summarised the context into a population-level statistic and discarded token-level detail. We ask whether this ``summarisation and forgetting'' can be derived from the attention mechanism itself, and answer in the affirmative.