FDE-Bench: Agents Achieve 55% Human Quality on Real Tasks
Baidu AI Cloud researchers introduced FDE-Bench, a benchmark for measuring how well AI agents handle underspecified, real-world business requests, according to a paper posted to arXiv on 10 October 2026.
The Challenge: Vague Requirements at Scale
The benchmark draws from 49 cases sourced from real customer engineering projects, covering combinatorial optimization and machine learning tasks. Unlike controlled lab settings, these requests arrived incomplete—missing requirements, unclear success criteria, and ambiguous scope. The benchmark tests whether agents can navigate this ambiguity the way human engineers do: by asking clarifying questions, iterating on solutions, and delivering working systems despite partial information.
Measured Performance Gap
Researchers evaluated 18 agent configurations and reported performance using a normalized delivery score where 1.0 represents a human engineer's reference solution. The highest mean delivery score achieved was 0.5510, indicating that even the best-performing agent configuration operated at roughly 55% of human engineer quality on average (arXiv).
When measuring outright delivery success—defined as planned runs that matched the human-engineer reference—the best result was 6.12% arXiv. In other words, even top-tier agent configurations succeeded in matching human-quality solutions in fewer than one in sixteen attempts.
Why This Matters for the Agent Economy
This benchmark addresses a critical gap in agent evaluation. Most benchmarks test isolated capabilities—answering factual questions, solving math problems, or generating code snippets. FDE-Bench instead measures end-to-end delivery: the agent's ability to take a messy, real business problem and ship a solution, including the harder human work of requirement negotiation and iterative refinement.
The results suggest that while agents excel at defined subtasks, they struggle with the disambiguation and adaptive problem-solving that characterize actual engineering work. A 6% success rate on unambiguous delivery, even for the best configuration, indicates a significant gap between agent capability and production-ready autonomy for complex, vaguely specified projects.
Open Benchmark and Code
The research team released both the benchmark dataset and evaluation code publicly, enabling other developers and researchers to test their own agents against these real-world cases. The benchmark is available at GitHub, allowing the agent development community to iterate toward higher delivery rates.
