Attention Once Is All You Need: Efficient Streaming Inference with Stateful Transformers

ArXi:2605.13784v1 Announce Type: new Conventional transformer inference engines are request-driven, paying an O(n) prefill cost on every query. In streaming workloads, where data arrives continuously and queries probe an ever-growing context, this cost is prohibitive. We