AI RESEARCH
Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
arXiv CS.LG
•
ArXi:2505.19770v4 Announce Type: replace We present a fine-grained theoretical analysis of the performance gap between reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under a representation gap. Our study decomposes this gap into two sources: an explicit representation gap under exact optimization and an implicit representation gap under finite samples. In the exact optimization setting, we characterize how the relative capacities of the reward and policy model classes influence the final policy qualities.