title: "FrontierCode 1.1: Benchmark Measures AI Code Against Maintainer Standa" slug: "frontiercode-11-benchmark-measures-ai-code-against-maintainer-standards" published: "2026-10-13" beat: "Research" tags: ["Research"] creator: "Agentry Newsroom" editor: "Susanne Sperling, Editor — Human in the Loop" tools: ["Claude (Anthropic)", "Perplexity Sonar"] creativeWorkStatus: "verified" dateReviewed: "2026-10-13" aiActArticle50: "compliant" humanView: "https://agentry.news/research/frontiercode-11-benchmark-measures-ai-code-against-maintainer-standards" agentView: "https://agentry.news/agent/frontiercode-11-benchmark-measures-ai-code-against-maintainer-standards"
Cognition released FrontierCode 1.1 in July 2026, a benchmark that evaluates whether AI-generated code changes would satisfy real open-source maintainers' criteria for merging. The evaluation, publish
Drafted by an AI agent. Verified by Susanne Sperling, Editor — Human in the Loop. AI policy.
Cognition announced FrontierCode in June 2026 and released version 1.1 in July 2026, introducing a benchmark designed to measure whether an AI agent's code changes would meet the acceptance criteria real open-source maintainers apply when deciding whether to merge pull requests Artificial Analysis.
Unlike benchmarks that test raw coding capability, FrontierCode 1.1 evaluates a concrete, shipping outcome: whether an agent's repository modifications—evaluated against deterministic tests, hidden tests, command checks, file and code-scope limits, and qualitative review—would satisfy merge criteria set by actual maintainers. The benchmark uses a scoring model where failing any blocking check produces zero; passing all blocking checks yields a score between 70% and 100% Artificial Analysis.
Cognition constructed the benchmark with maintainers from 36 open-source repositories, with each task requiring more than 40 hours of work to design and validate Artificial Analysis. The primary "FrontierCode 1.1 Main" evaluation contains 100 tasks drawn from a broader set of 150 total tasks; the tasks remain private to prevent benchmark contamination in model training Artificial Analysis.
This depth reflects a critical gap between lab benchmarks and production reality: a code change can pass automated tests and still fail to merge if it violates maintainer conventions, introduces technical debt, or misaligns with project architecture. FrontierCode 1.1 directly measures that gap.
Claude Opus 5.5 achieved the leading reported result, evaluated in the "max with fallback" configuration, with an average score of 52% and a pass rate of 58% Artificial Analysis. The benchmark's headline finding: "Leading models still fail more than 40% of the time."
This gap underscores why coding agents remain limited in autonomous repository contributions. Even state-of-the-art models struggle to generate code that maintainers would actually accept—a concrete measure of what agents can and cannot do in a core developer workflow.
FrontierCode 1.1 represents a shift in how agent capabilities are evaluated. Rather than testing capability in isolation ("Can the model write code?"), it measures actionability: can an agent produce work that flows directly into production? For developer-facing agents, that distinction is the difference between a research finding and a shipped product that delivers business value.
The benchmark is now available for evaluation against additional models and agent architectures, providing a quantitative standard for assessing progress in autonomous code contribution—a core use case for enterprise AI agents.