ParanoiaEval: Agents Over-Defend Code in 11%–59% of Tasks

Researchers submitted ParanoiaEval to arXiv on October 6, 2026, introducing a new benchmark for measuring when coding agents apply unnecessary defensive work arXiv. The evaluation framework includes 200 paired coding tasks designed to test whether agents can calibrate risk treatment appropriately or instead default to excessive caution.

What ParanoiaEval Measures

The benchmark, supported by a public GitHub repository ZhuoningXu/ParanoiaEval_release indexed October 4, 2026, isolates a blind spot in agent evaluation: the gap between coding capability and security judgment. Existing benchmarks typically measure whether agents can write correct code or fix bugs. ParanoiaEval instead asks whether agents know *when* defensive measures are necessary and when they are wasteful.

This distinction matters because agents that over-defend—adding input validation, logging, or error handling that the task does not require—waste tokens, slow execution, and can introduce complexity that itself becomes a liability.

Why This Matters for Agent Deployment

As organizations deploy coding agents to production codebases, the cost of unnecessary defensive work compounds. A single over-cautious agent decision multiplied across thousands of code generation runs can degrade throughput and increase maintenance burden. ParanoiaEval quantifies this trade-off by scoring agent outputs on both functional correctness and proportionality of defensive coverage.

The research identifies a critical decoupling: strong coding ability does not reliably predict appropriate defensive behavior. Agents that excel at algorithm implementation may still apply blanket security patterns regardless of context—a form of reward hacking that passes traditional correctness tests but fails real-world efficiency criteria.

Implications for Agent Evaluation

This finding aligns with broader efforts in the agent evaluation community to move beyond single-metric benchmarks. Earlier frameworks focused on success rates; ParanoiaEval introduces a second dimension: *proportionality under uncertainty*. The 11.2% to 58.7% range of unnecessary defensive work across test runs suggests high variance depending on model architecture, prompt framing, and task complexity.

For development teams building agent coding systems, the benchmark provides a concrete evaluation tool. For enterprises evaluating agent suppliers, ParanoiaEval offers a new lens on agent reliability—not just whether agents solve problems, but whether they solve them efficiently.

The research is immediately usable via the public repository, enabling researchers and engineers to test proprietary and open-source agents against the same paired task suite. Early adoption by benchmarking communities could accelerate adoption of proportionality testing in agent evaluation workflows.