Thinking into the Future: Latent Lookahead Training for Transformers

ArXi:2603.20219v1 Announce Type: cross Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time. Although very scalable, this objective forces the model to commit at every step, preventing it from exploring or reflecting upon multiple plausible continuations. Furthermore, the compute allocation across tokens is uniform; every token is formed based on a single forward-pass, potentially limiting the model's expressiveness in cases where difficult tokens require inherently compute. Towards addressing these limitations, we