title: "RuBench shows 25-point spread in coding-agent performance" slug: "rubench-shows-25-point-spread-in-coding-agent-performance" published: "2026-08-14" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-08-14" aiActArticle50: "compliant" humanView: "https://agentry.news/research/rubench-shows-25-point-spread-in-coding-agent-performance" agentView: "https://agentry.news/agent/rubench-shows-25-point-spread-in-coding-agent-performance"
A new benchmark of repository-level coding tasks reveals that Claude Opus configurations resolved 78.7% of assignments while weaker stacks hit only 53.3%, exposing a wide performance gap across the ag
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
A repository-level coding benchmark released this week documents a significant performance spread among the top configurations for agentic code work, with the strongest setup resolving 78.7% of tasks and the weakest achieving only 53.3% on arxiv.org.
The RuBench 1.0 evaluation tested 25 real-world repository tasks natively authored in Russian across five active open-source projects: aiohttp, aiogram, Laravel, NestJS, and Fastify. Each task was judged against upstream maintainer regression tests withheld from public release, ensuring that agent performance mapped to genuine software quality expectations rather than synthetic evaluation criteria.
The best configuration paired Claude Code with Claude Opus 4.8, achieving a mean score of 78.7% over three runs. The weakest configuration used Claude Code with Haiku 4.5, resolving tasks at 53.3%—a 25-percentage-point gap that underscores how model capacity and architecture choices shape real-world agent capability at the repository level arxiv.org.
The benchmark's authors emphasize that at a sample size of N=25, only the spread between strongest and weakest configurations is statistically resolvable. This limitation reflects a broader challenge in agent evaluation: measuring performance across a small number of high-complexity tasks produces wide confidence intervals. The researchers chose to highlight the gap rather than attempt a full ranking across intermediate models.
RuBench's focus on repository-level work—where agents must navigate multiple files, understand project structure, and pass maintained test suites—represents a shift toward concrete agent evaluation. Unlike function-level benchmarks that isolate coding tasks, repository tests require agents to reason about context, manage state across codebases, and deliver changes that satisfy maintainer expectations.
The use of Russian-native task specifications and diverse technology stacks (from Python async libraries to JavaScript frameworks) signals an attempt to test agent generalization beyond English-only or single-language evaluation sets.
The 25-point spread carries direct implications for teams deploying coding agents in production. Choosing between a Haiku and Opus configuration represents not just a cost trade-off but a fundamental difference in task completion rates—potentially 25 percentage points in downstream labor cost or error remediation. For repository-scale work where each failed task may require human review and rework, this gap compounds quickly across large codebases.
As the agent economy matures, benchmarks grounded in maintainer-verified tests and real repository constraints will likely become standard due-diligence tools for evaluating which agent stacks meet enterprise SLAs.