AI RESEARCH
Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
arXiv CS.LG
•
ArXi:2605.06387v1 Announce Type: new On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher feedback and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its standard advantage weighted policy gradient suffers from three structural weaknesses, including high variance updates, vanishing gradients in zero-advantage regions, and exploration bottlenecks when corrective signals are insufficient. We. therefore.