Please help improving a CPU-only inference speed
r/LocalLLaMA
•
Generative AI
Open Source AI
AI Research
This is a request for help for the people that want to use locally very large models on Q8 and better quanta at all costs, in my case the cost is inference speed. So I have a 512GB DDR4 ECC 2666 with a Threadripper Pro 3945WS that gives me ca. 5-7tok/second for MiniMax-2.7 with llama.cpp CPU backend. Yes, it probably feels like torture for the ADHD generation, but I'm using it for processing LARGE specs and planning, and it steers a Qwen-3.6-27B for implementation and testing. Of