UserProxyBench Benchmarks Agent-User Fidelity Gap

Researchers at Sarvam AI have published a new evaluation framework for measuring whether simulated users behave realistically when training and benchmarking LLM agents—a critical gap in agent development as systems move into enterprise deployment.

The work, titled *UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training*, was submitted to arXiv on September 29, 2026. Authors Ashish Jain and Armaan Sandhu introduce the User Fidelity Score, a metric that measures whether simulated users follow private instructions independent of whether the agent succeeds at its task.

The Fidelity Problem

Agent benchmarks rely on user simulations to create repeatable evaluation environments. But simulated users often behave differently from real humans—and those differences can skew both benchmarks and training outcomes. The UserProxyBench framework, which layers on top of the tau-bench family of benchmarks, quantifies this gap across seven distinct user proxies.

In a controlled evaluation with the agent held at GPT-5.5 across 375 enterprise tasks, changing the user proxy alone changed mean task reward by 15.2 points—a substantial variance driven entirely by simulation fidelity, not agent capability.

Violations and Failure Modes

More concerning: 24.4% of successful episodes contained a user-specification violation, meaning the simulated user provided information or took actions that contradicted private instructions, even when the agent ultimately completed its task. This creates a blind spot: a benchmark might rate an agent highly for achieving an outcome while the agent actually succeeded *despite* the user violating specifications.

The dominant failure pattern was premature disclosure, in which the simulated user volunteered information before the agent requested it. This behavior had a measurable interaction effect: among successful episodes, premature disclosure was associated with 1.06 fewer tool calls on average while the task reward remained unchanged. In other words, agents appeared more efficient when users handed them answers—a misleading training signal for real-world deployment.

Cost-Fidelity Frontier

The research maps a cost-fidelity frontier across the seven user proxies tested, showing that organizations face concrete tradeoffs: higher-fidelity simulations consume more resources but produce more reliable training signals. For enterprise adoption, this means benchmark results and training outcomes depend materially on which proxy an organization chooses—a finding that has immediate implications for how agent systems should be evaluated before production use.

The paper is available on arXiv and documents the first systematic evaluation of user proxy fidelity as a variable in agent training and benchmarking.