AI RESEARCH
Position: Science of AI Evaluation Requires Item-level Benchmark Data
arXiv CS.AI
•
ArXi:2604.03244v1 Announce Type: new AI evaluations have become the primary evidence for deploying generative AI systems across high-stakes domains. However, current evaluation paradigms often exhibit systemic validity failures. These issues, ranging from unjustified design choices to misaligned metrics, remain intractable without a principled framework for gathering validity evidence and conducting granular diagnostic analysis. In this position paper, we argue that item-level AI benchmark data is essential for establishing a rigorous science of AI evaluation.