Trace2Eval: Rubric feedback lifts coding-agent scores 10%
New evaluation framework targets coding-agent measurement
Researchers introduced Trace2Eval, a contextualized evaluation method for coding agents, in a paper submitted to arXiv on November 4, 2026 arXiv. The work addresses a gap in how developers and researchers measure agent performance in real-world coding tasks.
Key findings on rubric-based feedback
The core finding: rubric-based feedback produced a mean score gain of 10.05 percentage points across 19 model-harness configurations arXiv. The maximum reported gain reached 19.28 percentage points, observed in the DeepSeek-V4-Pro with Claude Code configuration.
These gains matter because coding agents increasingly handle production tasks—from code generation to debugging. Without reliable evaluation, teams lack visibility into whether agents are improving or which configurations work best for their use cases.
Methodology: traces, rubrics, and calibrated judges
Trace2Eval constructs benchmarks from real-world interaction trajectories, not synthetic test sets. The evaluation pipeline combines three elements: human-refined rubrics that define success criteria, calibrated judge agents that apply those rubrics consistently, and source-code and runtime evidence that grounds verdicts in observable outputs arXiv.
This approach differs from binary pass/fail metrics. By anchoring evaluation to actual development work—captured as traces of agent-developer interactions—the framework captures how agents behave in conditions closer to production than typical benchmarks.
Significance for the agent evaluation beat
Trace2Eval enters a crowded field of agent-evaluation research. The concrete finding—that structured rubric feedback reliably improves scores—suggests a practical path for teams building or deploying coding agents. The 19.28-point ceiling for top performers hints that configuration and feedback quality matter as much as model scale.
The paper's emphasis on runtime evidence and source-code evidence also signals a shift in how agent researchers think about evaluation: not as a black box score, but as a traceable, auditable process tied to what developers can actually inspect.
Next steps for the research community
Published results on a narrow set of configurations (19 total) leave open questions about generalization to other model families, harness designs, and coding tasks beyond those in the study. The arXiv record, updated November 5, 2026, does not yet report peer-review status or planned code/dataset release.
