AI RESEARCH
Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches
arXiv CS.LG
•
ArXi:2605.18825v1 Announce Type: new Prefix caching is a key optimization in Large Language Model (LLM) serving, reusing attention Key-Value (KV) states across requests with shared prompt prefixes to reduce expensive prefill computation. However, its benefit depends critically on the eviction policy as GPU memory is scarce, and existing policies such as LRU largely treat cached blocks uniformly. This view ignores a fundamental property of LLM prompts: not all tokens are equally worth caching.