Open-source single-GPU reproductions of Cartridges and STILL for neural KV-cache compaction
r/LocalLLaMA
•
AI Hardware
AI Research
I implemented two recent ideas for long-context inference / KV-cache compaction and open-sourced both reproductions: Cartridges: STILL: The goal was to make the ideas easy to inspect and run, with benchmark code and readable implementations instead of just paper/blog summaries.