Towards Apples to Apples for AI Evaluations: From Real-World Use Cases to Evaluation Scenarios

ArXi:2605.07986v1 Announce Type: cross AI measurement science has a wide variety of methodologies and measurements for comparing AI systems, resulting in what often appear to be "apples-to-oranges" comparisons across AI evaluations. To move toward "apples-to-apples" comparisons in real-world AI evaluations, this work advocates for methodological transparency in evaluation scenarios, operational grounding, and human-centered design (HCD) principles.