AI RESEARCH
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
arXiv CS.AI
•
ArXi:2605.12288v1 Announce Type: cross Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We.