Agent benchmarks fail to control cost and contamination jointly

A systematic review examining 259 primary studies has identified a critical gap in how AI agents are being evaluated: the 17 most prominent benchmarks fail to jointly control three key threats to validity—data contamination, nondeterminism, and execution cost—according to research published in *Artificial Intelligence Review* by Springer Springer.

The Evaluation Crisis

The paper, titled "Large Language Model Agent Evaluation and Benchmarking: a Systematic Survey, Meta-Taxonomy, and Critical Research Roadmap," conducted a PRISMA-ScR systematic mapping review with a living-review corpus containing 294 studies. The authors examined whether leading benchmarks—the standard tests used to measure agent capability—had implemented controls for three distinct validity threats that typically plague autonomous system evaluation.

The findings are stark: none of the 17 benchmarks examined provided evidence that contamination, nondeterminism, and execution cost were controlled jointly. The paper also reported that none of these 17 benchmarks produced a complete standardized run-cost record, recorded as 0/17 Springer.

This matters because agents increasingly operate in high-stakes domains—financial trading, healthcare recommendation, legal research, code generation. If benchmarks cannot reliably measure what agents actually do under controlled conditions, enterprise adoption decisions and safety claims rest on shaky ground.

What the Gaps Mean

Data contamination occurs when training data leaks into test sets, inflating performance metrics. Nondeterminism means agents produce variable results on identical inputs, making reproducibility impossible. Execution cost—measured in compute, tokens, or time—is often omitted from comparisons, allowing benchmarks to report speed or accuracy gains that vanish when real-world resource constraints apply.

The paper's finding that no single benchmark controls all three jointly suggests the evaluation ecosystem has optimized for speed and leaderboard appeal rather than rigorous, cost-aware measurement. This creates a perverse incentive: agents can score higher on existing benchmarks while performing worse in production.

A Roadmap for Accountability

Instead of reliance on isolated benchmarks, the authors propose a shift toward reliability-first and cost-aware evaluation infrastructure. This framework would standardize how execution costs are recorded and published, require reproducibility checks, and build evaluation systems designed to catch contamination and nondeterminism by design Springer.

For enterprise buyers, developers, and regulators, the study underscores a hard truth: a high benchmark score is not a reliable signal of agent quality. Until evaluation infrastructure matures, concrete deployed results—not published leaderboard positions—remain the only trustworthy measure of agent capability.

The paper's call for standardized, reproducible evaluation comes as agent products accelerate toward production deployment across banking, legal, and technology sectors, where evaluation rigor directly impacts risk.