AI RESEARCH
Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
arXiv CS.CL
•
ArXi:2509.24328v2 Announce Type: replace LLMs have low GPU efficiency and high latency due to autoregressive decoding. Speculative decoding (SD) mitigates this using a small draft model to speculatively generate multiple tokens, which are then verified in parallel by a target model. However, when speculation accuracy is low, the overhead from rejected tokens can offset the benefits, limiting SD's effectiveness, especially at large batch sizes.