LoLBench: Coding Agents Resolve Only 14% of Real-World Tasks
Researchers introduced LoLBench, a 100-task multilingual benchmark across 29 software systems in five domains, on October 15, 2026, documenting severe limitations in how today's coding agents handle large-scale production codebases arXiv.
Benchmark Scope and Performance Ceiling
The team evaluated 28 coding agents against real-world engineering challenges. The benchmark's tasks average approximately 5,000 words in specification length, operate on software systems averaging 2.4 million source lines of code, and require implementation pull requests that modify roughly 5,500 lines of code. Despite this scale, the best-performing agent resolved only 14% of tasks and achieved a 52.7% Fail-to-Pass pass rate arXiv—meaning agents could fix broken tests but struggled to implement new functionality from scratch.
The multilingual nature of LoLBench and its focus on 29 distinct systems across five domains position it as a rare evaluation capturing how agents perform on production-grade complexity rather than toy problems or isolated code snippets.
File Localization as Critical Failure Point
The researchers identified incomplete code localization as the dominant failure mode: agents routinely cannot narrow down which files to modify within massive codebases. When researchers provided reference-derived file trees paired with API specifications, resolved rates jumped 16–22 percentage points, reaching a maximum of 34% arXiv.
This finding has direct implications for agent tooling and deployment. It suggests that guidance systems—whether human-provided or agent-assisted—that reduce the search space for code location are necessary prerequisites for reliable agent performance on enterprise systems.
What the Data Means for the Agent Economy
A 14% baseline resolution rate on production tasks underscores a hard constraint: current agents cannot autonomously ship code across large systems without either human review, architectural simplification, or external hints about where to look. The 34% ceiling when aided by reference documentation indicates the problem is tractable but not yet solved at scale.
For enterprises evaluating agent adoption, LoLBench provides concrete, reproducible evidence of current agent capabilities—and their boundaries. For agent developers, the file-localization gap identifies a specific architectural target: improving how agents search and reason about code structure in unfamiliar systems.
The benchmark was updated on October 15, 2026, and is accompanied by research documentation establishing measurement methodology across tasks, agents, and system categories arXiv.