AI RESEARCH
wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment
arXiv CS.LG
•
ArXi:2603.07211v1 Announce Type: new Direct Preference Optimization (DPO) aligns large language models by optimizing pairwise preferences and has shown remarkable effectiveness as a simple and scalable alternative to RLHF. However, in practice, preference data are often noisy. Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting. While partially effective, these methods treat noisy samples as a homogeneous source of uncertainty and fail to distinguish between different noise types, leading to sub-optimal alignment robustness.