Pure C implementation of the TurboQuant paper (ICLR 2026) for KV cache compression in LLM inference.
r/LocalLLaMA
•
Generative AI
Open Source AI
AI Research
Pure C implementation of the TurboQuant paper (ICLR 2026) for KV cache compression in LLM inference. Key vectors compressed to 1 bit via randomized Hadamard transform + sign hashing. Attention via XOR + popcount. Values independently quantized to Q4 or Q2. Total K+V: 4.9x-7.1x compression on Gemma 3 4B, saving up to 3.7 GB at 32K context. 1-bit attention cosine = 0.634, matching the 2/pi theoretical limit. All NEON paths verified against scalar reference. ASan clean, 26 test suites. No external dependencies. submitted by /u/Suitable-Song-302 [link] [comments.