AI RESEARCH
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
arXiv CS.LG
•
ArXi:2604.20140v1 Announce Type: cross Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks. DPO optimizes for the likelihood of generating preferred over dispreferred responses in their entirety and lacks the granularity to provide feedback on subsections of many-step solutions typical of reasoning tasks.