AI benchmarks systematically ignore how humans disagree, Google study finds
The Decoder
•
Generative AI
A Google study finds that the standard three to five human raters per test example often aren't enough for reliable AI benchmarks, and that splitting your annotation budget the right way matters just as much as the budget itself.