AI RESEARCH
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
arXiv CS.AI
•
ArXi:2604.08865v1 Announce Type: new Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting