AI RESEARCH
Failure Modes of Maximum Entropy RLHF
arXiv CS.LG
•
ArXi:2509.20265v3 Announce Type: replace In this paper, we show that Simple Preference Optimization (SimPO) can be derived as Maximum Entropy Reinforcement Learning, providing a theoretical foundation for this reference-free method. Motivated by SimPO's strong performance in offline preference optimization, we investigate whether Maximum Entropy RL can achieve similar results in online RLHF settings.