Coding Agents Hit 91% Functional Tests but Only 25% Secure
A benchmark of six frontier coding agents published to arXiv on October 14, 2026 has exposed a critical reliability gap between functional correctness and security arXiv.
Researchers evaluated the agents on 450 tasks derived from 150 real vulnerabilities spanning seven programming languages and 24 CWE categories. The benchmark, titled "Last Secure Code Benchmark: Is the Code You Vibe Secure?", specified functionality requirements without mentioning security—then evaluated submissions against both hidden functional and hidden security tests arXiv.
The Performance Gap
Functional pass rates ranged from 73.6% to 91.3% across the six agents tested. The strongest performer, identified as Claude Fable 5, achieved the highest functional pass rate and also led in secure-code generation—but only 25.1% of its outputs were both functional and secure Dev.to. The five other agents achieved combined functional-and-secure rates of 16.4% to 24.7% arXiv.
This 65-point gap—between functional correctness and secure-plus-functional code—underscores a fundamental limitation in current frontier coding agents. An agent that writes working code does not reliably write *safe* code.
Why This Matters for Agent Deployment
The findings arrive as enterprises increasingly deploy coding agents for real-world development tasks. A functional agent that introduces vulnerabilities on three-quarters of security-sensitive tasks represents both an operational risk and a liability exposure. Organizations relying on these agents for production systems would need extensive post-generation review and remediation workflows—offsetting much of the time savings the agents promise.
The benchmark's methodology—hiding security requirements while measuring security outcomes—mirrors real-world conditions. Developers often optimize for feature delivery; vulnerabilities emerge downstream. The benchmark captures this dynamic by not cueing agents to prioritize security, then measuring whether they do anyway.
Implications for the Agent Economy
The results suggest that frontier coding agents remain far from fully autonomous code generation for security-critical domains. Enterprise adoption of these tools will likely require layered validation: automated security scanning, human code review, and runtime monitoring. The 25% secure-code ceiling indicates that agent assistance should complement, not replace, security-focused development practices.
The benchmark's release on arXiv makes the full evaluation methodology public, enabling other teams to test new agents and variants as they ship. Future versions may incorporate post-hoc security prompting, retrieval-augmented code generation, or security-specific fine-tuning—all potential attack vectors for improvement.