AI RESEARCH
Why Global LLM Leaderboards Are Misleading: Small Portfolios for Heterogeneous Supervised ML
arXiv CS.LG
•
ArXi:2605.06656v1 Announce Type: new Ranking LLMs via pairwise human feedback underpins current leaderboards for open-ended tasks, such as creative writing and problem-solving. We analyze ~89K comparisons in 116 languages from 52 LLMs from Arena, and show that the best-fit global Bradley-Terry (BT) ranking is misleading. Nearly 2/3 of the decisive votes cancel out, and even the top 50 models according to the global BT ranking are statistically indistinguishable (pairwise win probabilities are at most 0.53 within the top 50 models.