AI RESEARCH
From Interpretability to Performance: Optimizing Retrieval Heads for Long-Context Language Models
arXiv CS.CL
•
ArXi:2601.11020v3 Announce Type: replace Advances in mechanistic interpretability have identified special attention heads, known as retrieval heads, that are responsible for retrieving information from the context. However, the role of these retrieval heads in improving model performance remains unexplored. This work investigates whether retrieval heads can be leveraged to enhance the long-context capabilities of LLMs. Specifically, we propose RetMask, a method that generates