AI RESEARCH
MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference
arXiv CS.CL
•
ArXi:2603.26557v1 Announce Type: new Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant ing information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model.