AI RESEARCH
PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
arXiv CS.LG
•
ArXi:2605.08581v1 Announce Type: new Modern online large language model (LLM) services, such as Retrieval-Augmented Generation (RAG) and agent systems, increasingly expose two prominent characteristics: prompt segmentation (e.g., system instructions, retrieved passages, tool outputs) and hotspot skew, where a small set of these segments recurs frequently across user requests. Failing to jointly exploit these patterns could lead to repeated prefill of hot segments and prolonged TTFT, undermining both throughput and user-perceived responsiveness.