PACE cuts agent evaluation cost to 1% with proxy benchmark

Researchers published "PACE: A Proxy for Agentic Capability Evaluation" on arXiv, introducing a method to predict AI agent benchmark scores using lightweight proxy evaluations instead of expensive full test runs arXiv.

How PACE Works

PACE constructs proxy benchmarks by selecting instances from existing non-agentic evaluations and fitting a regression model from the compact subset to target agentic benchmark scores arXiv. The approach treats agent capability evaluation as a prediction problem: rather than run full agentic test suites, developers can use smaller, cheaper non-agentic benchmarks to estimate performance.

Experimental Results

The researchers tested PACE across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks, achieving leave-one-out cross-validation mean absolute error under 4% while costing much less than 1% of full agentic evaluation arXiv. This accuracy-to-cost ratio addresses a concrete pain point: as AI labs scale agent testing, full benchmark suites become prohibitively expensive to run repeatedly during development cycles.

The paper also produced PACE-Bench, a concrete proxy benchmark dataset available for public use Hugging Face, enabling developers to apply the method without rebuilding infrastructure from scratch.

Market Implications

For the AI agent economy, faster and cheaper evaluation cycles reduce friction in agent product iteration. Teams can validate capability improvements or regressions across model checkpoints without waiting days for full benchmark runs or burning compute budgets. This directly affects time-to-market for agent releases and enterprise adoption timelines, where clients increasingly demand performance guarantees before deployment.

The research also surfaces a structural bottleneck: as agentic benchmarks become more complex—testing long-horizon planning, tool use, and error recovery—the cost gap between proxy and full evaluation widens. PACE's sub-4% error suggests that much of agent capability variance is captured by simpler, cheaper signals, a finding that could reshape how labs allocate evaluation resources.

Availability

The paper and PACE-Bench dataset are available for immediate use, making this a release-ready research contribution rather than a roadmap proposal arXiv Hugging Face.