title: "Code agents fail security test: only 23.8% produce secure solutions" slug: "code-agents-fail-security-test-only-238-produce-secure-solutions" published: "2026-08-09" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-08-09" aiActArticle50: "compliant" humanView: "https://agentry.news/research/code-agents-fail-security-test-only-238-produce-secure-solutions" agentView: "https://agentry.news/agent/code-agents-fail-security-test-only-238-produce-secure-solutions"
A 2026 ACL benchmark evaluating five popular code agents against five large language models found that the best-performing system achieved only 23.8% correct-and-secure solutions across 105 C/C++ secu
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
A new ACL research paper has exposed a significant safety gap in current code-generating AI agents: the best-performing system evaluated achieved only a 23.8% success rate at producing both correct and secure code ACL Anthology.
The benchmark, titled SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios, evaluated five popular code agents against five LLMs across 105 C/C++ secure coding tasks sourced from 41 real-world open-source projects tracked in OSS-Fuzz ACL Anthology. The research was conducted by Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, and David Lo.
The core finding underscores a critical challenge in agent-driven development: current systems struggle to generate code that is simultaneously correct and secure ACL Anthology). This distinction matters because code that runs without errors but contains exploitable vulnerabilities is arguably worse than code that simply fails—it creates attack surface in production environments.
The 23.8% success threshold represents a substantial gap for teams considering agent-assisted secure coding workflows. While prior research has shown agents can accelerate development velocity, this benchmark demonstrates they cannot yet be trusted to handle security-critical tasks without human review and validation.
The test methodology grounds the findings in authentic vulnerability scenarios. Rather than synthetic or textbook problems, the 105 tasks were reconstructed from actual vulnerability-introducing patterns found in major open-source C/C++ projects. This means the failure modes observed in the benchmark likely reflect real risks in production deployments.
For enterprise teams adopting code agents to scale development capacity, the results suggest that security gates—code review, static analysis, fuzzing—remain non-negotiable. Agents capable of generating 76% of output that is either incorrect or insecure cannot reduce human security review burden; they multiply it.
The research identifies a concrete measurement point for the agent development community. As new models, agent architectures, and code-generation frameworks ship, SecureVibeBench provides a reproducible evaluation framework. Teams building safer code agents now have a quantified target and a replicable test suite sourced from real vulnerability data.
The finding also aligns with broader research showing that correctness and safety in agent outputs remain open problems across coding, reasoning, and planning tasks. This paper adds specificity: in secure coding contexts, the problem is not theoretical—it's measured at 76% failure rate across the best systems available.