Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks

ArXi:2605.10639v1 Announce Type: new The rapid adoption of LLMs in both research and industry highlights the challenges of deploying them safely and reveals a gap in the systematic evaluation of toxicity benchmarks. As organizations increasingly rely on these benchmarks to certify models for customer-facing applications and automated moderation, unrecognized evaluation biases could lead to the deployment of vulnerable or unsafe systems.