ExpertCrowd-SWE Benchmark Tests Seven Coding Agents
A study titled ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases was posted to arXiv on November 9, 2026, and updated two days later (arXiv). The research evaluated seven frontier AI models on their ability to complete real-world software engineering work, offering one of the more granular public comparisons of coding agents to date.
What the study measured
Researchers tested the models against 100 expert-authored, multi-file software engineering tasks drawn from 87 open-source projects spanning 11 programming languages, according to the paper (arXiv). Each model was run five times per task, producing 3,495 scored runs in total — a sample size designed to capture variance in agent performance rather than rely on single-shot results, as reported by Yahoo Tech (Yahoo Tech).
Top results, narrow margin
Claude Opus 5.5, built by Anthropic, posted the highest score at 67.3%, according to the study (arXiv). GPT-6 Astra, built by OpenAI, followed at 63.7%, a gap of roughly 3.6 percentage points. But the study's authors found this difference was not statistically significant: the paired 95% confidence interval for the gap ran from −1.4 to +9.7 percentage points, with a p-value of 0.20, meaning the true difference could run in either direction or be negligible, as detailed in coverage of the paper (Datacamp).
That statistical caveat complicates any simple narrative of one model beating the other outright. Reporting on the study noted the results instead point to roughly four performance tiers among the seven models tested, rather than a single clear leader, based on analysis of the paper's findings (Becryptonews).
Why the methodology matters
Unlike many coding benchmarks that reuse familiar repositories or synthetic problems, ExpertCrowd-SWE drew its tasks from unfamiliar codebases authored by domain experts, a design intended to better simulate what engineering teams actually ask of coding agents in production settings, according to coverage of the study (ABC17 News). The breadth of languages and projects — 87 repositories across 11 languages — also sets it apart from narrower single-language benchmarks common in prior agent evaluations.
The study does not report any regulatory action, legal proceeding, or financial penalty tied to its findings; its contribution is purely empirical, offering engineering teams a more statistically rigorous lens on where frontier coding agents currently stand and where their performance remains too close to call.


