AI RESEARCH
From Raw Corpora to Domain Benchmarks: Automated Evaluation of LLM Domain Expertise
arXiv CS.CL
•
ArXi:2506.07658v3 Announce Type: replace Accurate domain-specific benchmarking of LLMs is essential, specifically in domains with direct implications for humans, such as law, healthcare, and education. However, existing benchmarks are documented to be contaminated and are based on multiple-choice questions, which suffer from inherent biases. To measure domain-specific knowledge in LLMs, we present a deterministic pipeline that transforms raw domain corpora into completion-style benchmarks without relying on other LLMs or costly human annotation.