AI RESEARCH
CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
arXiv CS.LG
•
ArXi:2605.11504v1 Announce Type: new Recent advances in Large Language Models (LLMs) have enabled agentic systems for complex, multi-step tasks; cybersecurity is emerging as a prominent application. To evaluate such agents, researchers widely adopt Capture The Flag (CTF) benchmarks. However, current CTF benchmarks reuse existing challenges, which exposes them to data contamination and potential cheating. Notably, we confirmed these issues in practice by integrating web search tools into an existing agent.