AI RESEARCH
Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation
arXiv CS.AI
•
ArXi:2603.10072v1 Announce Type: cross Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality.