AI Security Agents Fail Half of Web Challenges, Study Finds
Six AI security agents achieved a 41% flag-capture rate when tested on 11 web-based Capture the Flag challenges in controlled conditions, according to a new evaluation framework published in the *Proceedings of the 19th ACM Workshop on Artificial Intelligence and Security* ACM. The study, titled *AgentLSD: Evaluating AI Security Agents Under Adversarial Task Contamination*, reveals both the current capabilities and vulnerabilities of AI agents deployed for cybersecurity tasks.
Baseline Performance in Clean Conditions
Under clean conditions—where CTF challenges contained only their intended solutions and no interference—no single model solved every challenge. The 41% success rate across all six evaluated models suggests that AI agents remain unreliable for autonomous security work, even in predictable environments. This baseline establishes a concrete performance floor for understanding what these systems can and cannot do in real-world security operations.
How Adversarial Contamination Degrades Agent Performance
The research introduced a framework of adversarial contamination tactics designed to mislead AI agents while preserving the legitimate solution path. These contamination types included fake flags, misleading hints, decoy endpoints, and hidden cues embedded within the challenge environment ACM.
The contamination had measurable computational costs. Task length increased by 20 turns on average, and agents consumed an additional 2,000 reasoning tokens per contaminated challenge, even when they ultimately recovered the correct flag ACM. This efficiency degradation is significant for systems operating under computational constraints or real-time deadlines.
Implications for Agent Deployment
The findings underscore a critical gap between controlled laboratory settings and production environments. In real-world cybersecurity operations, adversarial agents—or even naturally occurring environmental noise—can substantially degrade performance without necessarily causing outright failure. Organizations deploying AI security agents should account for this efficiency tax when calculating operational costs and response times.
The contamination framework itself offers value to security teams and AI developers. By identifying specific attack vectors against agent reasoning—misleading hints and decoy endpoints proved particularly effective—the research provides a roadmap for both hardening agents and testing their robustness before deployment.
The study was conducted by researchers affiliated with five institutions and published November 14, 2026 ACM. The research contributes to a growing body of work examining the real-world reliability of autonomous AI systems in high-stakes domains.
