---
title: "RuBench shows 25-point spread in coding-agent performance"
slug: "rubench-shows-25-point-spread-in-coding-agent-performance"
published: "2026-08-14"
beat: "Research"
tags: ["Research"]
creator: "Agentry Newsroom"
editor: "Susanne Sperling, Editor — Human in the Loop"
tools: ["Claude (Anthropic)", "Perplexity Sonar"]
creativeWorkStatus: "verified"
dateReviewed: "2026-08-14"
aiActArticle50: "compliant"
humanView: "https://agentry.news/research/rubench-shows-25-point-spread-in-coding-agent-performance"
agentView: "https://agentry.news/agent/rubench-shows-25-point-spread-in-coding-agent-performance"
---# RuBench shows 25-point spread in coding-agent performance

> A new benchmark of repository-level coding tasks reveals that Claude Opus configurations resolved 78.7% of assignments while weaker stacks hit only 53.3%, exposing a wide performance gap across the ag

*Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. [AI policy](/ai-policy).*

A repository-level coding benchmark released this week documents a significant performance spread among the top configurations for agentic code work, with the strongest setup resolving 78.7% of tasks and the weakest achieving only 53.3% on [arxiv.org](https://arxiv.org/html/2607.06411v2).

The **RuBench 1.0** evaluation tested 25 real-world repository tasks natively authored in Russian across five active open-source projects: aiohttp, aiogram, Laravel, NestJS, and Fastify. Each task was judged against **upstream maintainer regression tests** withheld from public release, ensuring that agent performance mapped to genuine software quality expectations rather than synthetic evaluation criteria.

## Best vs. Weakest Configurations

The **best configuration** paired **Claude Code with Claude Opus 4.8**, achieving a mean score of 78.7% over three runs. The **weakest configuration** used **Claude Code with Haiku 4.5**, resolving tasks at 53.3%—a **25-percentage-point gap** that underscores how model capacity and architecture choices shape real-world agent capability at the repository level [arxiv.org](https://arxiv.org/html/2607.06411v2).

The benchmark's authors emphasize that at a sample size of N=25, only the spread between strongest and weakest configurations is statistically resolvable. This limitation reflects a broader challenge in agent evaluation: measuring performance across a small number of high-complexity tasks produces wide confidence intervals. The researchers chose to highlight the gap rather than attempt a full ranking across intermediate models.

## Task Design and Real-World Relevance

RuBench's focus on repository-level work—where agents must navigate multiple files, understand project structure, and pass maintained test suites—represents a shift toward concrete agent evaluation. Unlike function-level benchmarks that isolate coding tasks, repository tests require agents to reason about context, manage state across codebases, and deliver changes that satisfy maintainer expectations.

The use of Russian-native task specifications and diverse technology stacks (from Python async libraries to JavaScript frameworks) signals an attempt to test agent generalization beyond English-only or single-language evaluation sets.

## Implications for Agent Builders

The 25-point spread carries direct implications for teams deploying coding agents in production. Choosing between a Haiku and Opus configuration represents not just a cost trade-off but a fundamental difference in task completion rates—potentially 25 percentage points in downstream labor cost or error remediation. For repository-scale work where each failed task may require human review and rework, this gap compounds quickly across large codebases.

As the agent economy matures, benchmarks grounded in maintainer-verified tests and real repository constraints will likely become standard due-diligence tools for evaluating which agent stacks meet enterprise SLAs.