nano-KvLLM: Integrating KV Cache Compression into nano-vLLM for Long-Context Inference

Hi everyone, I recently built nano-KvLLM, an easy-to-use lightweight inference framework based on nano-vLLM for efficient KV-cache management in LLM serving. Github: A key goal of this framework is to preserve the original nano-vLLM code layout as much as possible, with only simple and minimal modifications, so that users can easily learn from the codebase and develop their own extensions on top of it. Right now, nano-KvLLM already s KV-cache compression in the nano-vLLM execution pipeline. Users can quickly plug in and test their own compression methods, or build on top of the built-in.