AgentPerfBench benchmarks agentic LLM inference performance

Researchers released AgentPerfBench, a new benchmark suite designed to measure inference performance across agentic large language models, according to an arXiv paper published September 28 arXiv. The work, authored by Cheuk Hang Lau, Zeyu Cao, Kevin Wong Cheuk Yin, and colleagues, addresses a gap in agent evaluation by grounding performance metrics in real-world traces rather than synthetic workloads.

Real-World Agent Traces

The suite draws directly from established agentic benchmarks, including SWE-Bench (software engineering task completion) and TerminalBench (command-line agent execution), ensuring that performance measurements reflect authentic agent behavior patterns. This grounding in real traces distinguishes AgentPerfBench from inference benchmarks that rely on generic prompt patterns, allowing researchers and practitioners to understand how inference bottlenecks manifest when agents navigate complex multi-step tasks.

Why Inference Performance Matters for Agents

As agentic systems move beyond single-turn completions into iterative planning and tool-use workflows, inference latency and throughput become critical operational constraints. An agent that reasons correctly but exceeds token-per-second budgets cannot scale in production environments. AgentPerfBench directly measures these constraints using traces captured from real benchmarks, providing a concrete foundation for comparing model efficiency across different agentic architectures and inference hardware configurations.

Benchmark Scope

The paper's methodology leverages task traces from SWE-Bench, which evaluates agents on software engineering challenges including code navigation, refactoring, and bug repair, and TerminalBench, which tests agents' ability to complete shell-based system administration and data processing tasks. By instrumenting these benchmarks to capture agent inference patterns—token counts, latency distributions, tool-call frequencies—the authors created a repeatable evaluation framework that spans multiple agent classes and use cases.

Broader Context

The timing of AgentPerfBench reflects growing maturity in the agent evaluation ecosystem. As enterprises deploy production agent systems and startups ship agentic products, the ability to benchmark and optimize inference performance becomes a competitive and operational necessity. The paper contributes concrete measurement tools for the community to track these capabilities across model releases and inference platforms.

AgentPerfBench is available through arXiv's public repository and can serve as a reference implementation for teams building or selecting agentic inference systems arXiv.