PACE proxy cuts agentic eval cost to 1% of full suite
Researchers publish PACE proxy to slash agentic evaluation cost
A new arXiv paper proposes PACE, a method for predicting how well language models perform on agentic tasks without running expensive full evaluations arXiv. The technique constructs lightweight proxy benchmarks by selecting instances from existing non-agentic evaluations and fitting regression models to map those scores onto agentic benchmark performance.
The approach addresses a real friction point in the agent economy: running comprehensive agentic benchmarks—which require agents to execute multi-step tasks in real or simulated environments—is computationally expensive and time-consuming. PACE promises to unlock faster model comparison and ranking cycles for teams shipping agent products and infrastructure.
Benchmark results across 14 models
The PACE paper reports validation across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks arXiv. The core findings: leave-one-out cross-validation MAE under 4%, Spearman rank correlation above 0.80, and pairwise model-ranking accuracy around 85%. These metrics suggest the proxy accurately estimates which models will rank highest on true agentic tasks.
The method's cost profile is the headline draw: PACE evaluation runs at less than 1% of the cost of full agentic evaluation arXiv. For teams iterating on model selection, safety testing, or capability benchmarking, that efficiency unlock could compress evaluation cycles from weeks to hours.
Why this matters for the agent economy
Agent builders today face a choice: spend heavily on comprehensive benchmarking to trust a model's real-world capability, or make quick calls on limited signals. PACE sits between those poles—fast enough to use in pre-selection workflows, accurate enough to avoid ranking surprises downstream. The paper does not claim proxy results should replace full agentic evaluation for production safety gates; rather, it positions PACE as a filter to shrink the candidate pool before expensive final testing.
The work also opens a research direction: as more agentic benchmarks stabilize and scale, proxy construction should improve. Right now PACE is bounded by the diversity and quality of non-agentic benchmarks available; future versions could absorb richer signal from task families closer to real agent work.
Code and datasets are available via HuggingFace HuggingFace, enabling developers to replicate the method or apply it to proprietary model lineups.