Safety scores don't predict agent jailbreak risk, study finds

Researchers including Md Jueal Mia, Yanzhao Wu, and Selcuk Uluagac published findings in September 2026 showing that high safety scores on large language models do not reliably predict resistance to jailbreaks in agentic systems arXiv. The paper, titled "SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration," challenges a widespread assumption in the AI safety community: that demonstrable native alignment translates directly to robustness against adversarial manipulation.

The Gap Between Alignment and Robustness

The study's core finding is counterintuitive for practitioners building agent safety frameworks: strong native alignment does not necessarily imply robustness to adversarial jailbreaks. This distinction matters because agents operate across multiple stages—planning, reasoning, tool selection, memory retrieval, and execution—each representing a potential attack surface that traditional safety evaluations may miss.

Where conventional safety benchmarks measure final-response safety (whether an agent outputs harmful content), jailbreak vulnerabilities can manifest in how an agent *plans* to achieve a user's request, how it *uses tools* to execute unauthorized actions, or how it *retrieves and weights* past instructions against current safeguards. An agent might refuse a direct harmful request while still being manipulated into achieving the same outcome through multi-step reasoning or tool chaining.

Rethinking Evaluation Methodology

The researchers concluded that agent safety evaluations must broaden beyond endpoint assessment. Rather than treating an agent's final response as the only meaningful safety signal, evaluations should examine intermediate behavior across planning, memory, tool use, and reasoning chains. This shift reflects a fundamental change in how AI systems operate: agents are no longer passive responders but active problem-solvers capable of decomposing tasks, maintaining state, and leveraging external tools—each step a potential vulnerability.

The implications extend across the agent development stack. Companies shipping agents to production, developers building agentic frameworks, and security researchers designing defenses all operate under incomplete threat models if they rely solely on traditional safety scores. An agent that scores highly on safety benchmarks may still be vulnerable to sophisticated attacks that exploit the agentic architecture itself.

Practical Takeaways for Builders

The study underscores why agent-specific threat models are necessary. As the industry moves toward autonomous systems handling high-stakes tasks—financial decisions, legal work, medical research—the gap between alignment and robustness becomes a material risk. Organizations deploying agents need to conduct their own intermediate-behavior audits rather than assuming that an upstream model's safety certification carries through to agentic deployment.

The research aligns with a growing consensus that agentic AI requires new evaluation and defense frameworks, distinct from those designed for static language models.