title: "CWE-bench: Agents Reach 68% on Security Repair Tasks" slug: "cwe-bench-agents-reach-68-on-security-repair-tasks" published: "2026-10-17" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-17" aiActArticle50: "compliant" humanView: "https://agentry.news/research/cwe-bench-agents-reach-68-on-security-repair-tasks" agentView: "https://agentry.news/agent/cwe-bench-agents-reach-68-on-security-repair-tasks"
Researchers at several institutions evaluated 11 frontier AI models on a new benchmark spanning 120 code-repair tasks across 73 vulnerability types, finding Grok 4.7 and GPT-6 Astra tied at 68% pass@1
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Researchers posted CWE-bench, a new security evaluation framework, to arXiv on October 12, 2026, measuring how well frontier coding agents can audit and patch vulnerabilities in real-world software repositories arXiv.
The benchmark comprises 120 held-out audit-and-patch tasks covering 73 Common Weakness Enumerations and the 10 OWASP Top 10 2025 categories across multiple programming languages, according to the benchmark's published summary Benchmark List. This represents a shift in how security research evaluates AI agents—focusing on defense and remediation rather than exploitation.
The research team, led by Gonzalo Gonzalez-Pumariega, Ajay Jagannath, Adit Jain, Guram Gogia, Soumyadeep Bakshi, and Nazneen Rajani, evaluated 11 models in agentic harnesses. The leading systems—Grok 4.7 and GPT-6 Astra—tied at 68% pass@1, meaning they successfully repaired vulnerabilities on a single attempt in two-thirds of cases Collinear Blog.
When permitted multiple attempts, Grok 4.7 achieved the highest performance, reaching 81% pass@4—indicating that on four tries, the model successfully completed four out of five repair tasks Benchmark List.
The results signal both progress and gaps in agent-assisted security workflows. A 68% success rate on first attempt suggests that agents can meaningfully reduce the manual burden of vulnerability remediation, particularly in large codebases where human auditors would otherwise review thousands of potential issues. However, the non-trivial failure rate underscores why human oversight remains critical in production environments.
The benchmark's structure—combining multiple vulnerability classes and programming languages—mirrors real-world conditions more closely than narrow, single-language evaluations. This concrete grounding distinguishes CWE-bench from purely theoretical agent capability studies.
The research was updated October 14, 2026, and is available in full at arXiv, making the benchmark tasks and evaluation protocol transparent to the broader security and AI research community arXiv.