llama.cpp speculative checkpointing was merged
r/LocalLLaMA
•
Generative AI
Open Source AI
Some prompts get a speedup, others don't (cases of low draft acceptance streak). Good working params depend on the task type and repetition patterns. For coding, I got some 0%~50% speedup with these params: --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64 submitted by /u/AdamDhahabi [link] [comments]