AI RESEARCH
Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
arXiv CS.AI
•
ArXi:2605.19461v1 Announce Type: new On-policy reinforcement learning methods like GRPO suffer from mode collapse: they exhibit reduced solution diversity, concentrating probability mass on a single solution once discovered and ceasing exploration of alternative strategies. We show this stems from reverse KL minimization's mode-seeking behavior, which reinforces the first high-reward trajectory found rather than maintaining a distribution over multiple diverse solutions.