RuBench: Claude Opus reaches 78.7% on Russian repo tasks
A repository-level coding benchmark published on arXiv found that Claude Code with Opus 4.8 resolved 78.7% of real maintenance tasks drawn from live Russian-language open-source projects, marking concrete progress in evaluating deployed agent performance on substantive, non-synthetic work arXiv.
The benchmark, called RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications, tested agents against 25 maintenance tasks mined from five active repositories: aiohttp, aiogram, Laravel, NestJS, and Fastify arXiv. The task specifications were natively written in Russian rather than translated, reflecting how developers actually describe and track work in non-English-speaking communities.
Researchers evaluated multiple deployed configurations—Claude Code with Opus 4.8, Sonnet 5, and Haiku 4.5, alongside Codex CLI with GPT-5.5—running three independent trials per configuration arXiv. Opus 4.8 performed strongest at 78.7% pass@1, the rate at which an agent completed a task correctly on the first attempt without iteration.
Why Repository-Level Tasks Matter
Maintenance tasks grounded in actual commit histories differ sharply from synthetic benchmarks. They require agents to understand existing codebase context, trace dependencies, and apply fixes that integrate seamlessly with live projects. A task might involve reviewing recent commits, diagnosing a bug's root cause, and generating a patch—work that demands both code understanding and real-world constraints.
The benchmark's construction from recent fix commits in production repositories means agents faced the same complexity and ambiguity that human developers encounter. This grounds the 78.7% result in practical capability rather than controlled evaluation conditions.
Caveat: Small Sample Size
The authors explicitly note that at N=25 tasks, only gaps separating the strongest performers from the weakest were statistically resolvable arXiv. The benchmark's small size limits confidence in fine-grained performance comparisons, though it does establish that modern production agents succeed on three-quarters of real repository work in this language and domain.
The result contributes to a growing body of evidence that code agents are moving beyond toy problems. Where benchmarks like ReviewBench and ScoringBench have tested narrower aspects of code review and evaluation, RuBench anchors capability measurement in full-context repository maintenance—closer to what enterprises and open-source teams actually deploy agents to solve.
For teams evaluating whether to invest in agent-assisted development workflows, the 78.7% pass rate on authentic Russian-language tasks suggests that deployed systems are ready for real work—though still requiring human review and iteration on the one-quarter of cases where agents falter.