AI RESEARCH
Continuous Semantic Caching for Low-Cost LLM Serving
arXiv CS.LG
•
ArXi:2604.20021v1 Announce Type: new As Large Language Models (LLMs) become increasingly popular, caching responses so that they can be reused by users with semantically similar queries has become a vital strategy for reducing inference costs and latency. Existing caching frameworks have proposed to decide which query responses to cache by assuming a finite, known universe of discrete queries and learning their serving costs and arrival probabilities.