Coding agents pass most criteria but complete few tasks

Researchers released Co-Design Bench, a multimodal collaborative coding evaluation benchmark, on October 26, 2026, exposing a gap between how coding agents handle partial requirements and how often they ship complete, working solutions.

Benchmark Design and Scale

The benchmark comprises 273 tasks—100 multimodal editing tasks and 173 repository editing tasks—derived from 37,463 deduplicated queries across 26,314 real user coding sessions. Researchers constructed the evaluation using a shared agentic harness to test ten model endpoints under consistent conditions, ensuring that performance differences reflect genuine capability gaps rather than setup variability.

Key Finding: Criteria vs. Completion

The strongest endpoint tested, gpt-6-astra, achieved an average pass rate of 69.9% of acceptance criteria per task, but fully satisfied only 26.4% of tasks. This split reveals a critical distinction in agent performance: agents often make measurable progress on code work—fixing bugs, adding features, meeting partial specifications—without delivering production-ready, end-to-end solutions.

What This Means for Real-World Deployment

The gap between partial success and full task completion has immediate implications for enterprises considering coding agents. A 69.9% criteria pass rate might appear strong on paper; it suggests agents are generally competent at individual requirements. However, a 26.4% task-completion rate indicates that in roughly three out of four cases, human review or intervention is likely needed before code can merge to production.

This pattern aligns with how coding agents currently operate in practice: they excel at generating boilerplate, refactoring, and isolated fixes, but struggle with the systemic constraints of a real repository—test coverage, integration points, style enforcement, and deployment readiness.

Benchmark Transparency and Replicability

The dataset documentation, evaluation methodology, and evaluation submissions are publicly available, enabling independent replication and future model comparisons. The benchmark is grounded in authentic user behavior rather than synthetic or curated task sets, strengthening its relevance to real deployment scenarios.

As coding agents mature, benchmarks that distinguish between partial and full task success will become increasingly important for procurement decisions and capability claims.