OpenHarmony benchmark: build success near 100%, task completion stalls

Researchers evaluating LLMs and coding agents on OpenHarmony reported a striking divergence between build reliability and task completion, according to a preprint posted on arXiv and updated August 28, 2026. Mean final build success rates ranged from 94.77% to 100.00%, yet mean task completion remained far lower at 48.36% to 58.39%.

The Saturation Problem

The OpenHarmony benchmark, designed to evaluate coding agents in real-world conditions, shows that agent systems have largely solved the mechanical problem of compilation and binary success. Build success rates near saturation suggest that modern LLM-based coding agents can reliably produce syntactically correct, executable code—a floor expectation for production use arXiv.

This closure of the build-success gap reflects maturation in prompt engineering, in-context learning, and underlying model capability over the past 18 months. Agents no longer routinely fail at the syntax level.

Where Agents Still Falter

Task completion—the percentage of assigned work that agents finish correctly—lags by roughly 40 percentage points. This gap points to a second-order problem: agents can write correct code, but struggle with multi-step reasoning, error recovery, requirement inference, and planning across long horizons.

A task completion ceiling around 50% suggests agents excel at isolated, well-specified subtasks but break down when required to sequence decisions, handle ambiguity, or backtrack after intermediate failures. The benchmark does not isolate whether failures stem from the agent's planning capability, the underlying model's reasoning depth, or inadequate tooling for state recovery.

Implications for Enterprise Adoption

For organizations considering agent-based coding pipelines, the data cuts both ways. Near-perfect build success reduces the risk of deploying agent-generated code that crashes immediately. However, task completion below 60% means agents will require human review, specification refinement, or hybrid workflows in which humans and agents share responsibility.

This profile—reliable execution of understood tasks, inconsistent completion of complex ones—mirrors the current state of autonomous systems across robotics and logistics. Success requires pairing agents with robust feedback loops, clear APIs, and human escalation paths.

Next Steps

The OpenHarmony results, reported in the abstract of the evaluation paper, suggest that the next frontier for agent research lies not in build reliability but in planning, decomposition, and error handling. Teams investing in agent infrastructure should prioritize observability, rollback mechanisms, and partial-task isolation over binary success metrics.