MLOps-Bench: Agents solve just 31% of real ML engineering tasks

A Production-Scale Test for AI Agents

Researchers introduced MLOps-Bench, a repository-grounded benchmark designed to measure whether AI agents can handle the concrete challenges of production machine-learning engineering. Published October 16, 2026 and updated October 19, 2026, the benchmark comprises 212 validated tasks extracted from 31 public machine-learning repositories, covering data pipelines, feature engineering, model training, serving, monitoring, CI/CD, governance, and integration workflows.

The benchmark represents a shift away from abstract capability tests. Rather than evaluating agents on synthetic or isolated coding challenges, the researchers sourced real problems from established open-source ML projects—work that actual engineers face in production environments.

Six Agents Evaluated; One Leads

The study evaluated six agent configurations against these tasks. The best result came from **Codex using GPT-5.5**, which completed 31.2% of tasks and passed 68.7% of individual tests. The gap between test passage and task completion—37.5 percentage points—signals a crucial distinction: agents may solve isolated components within a workflow but struggle when those components must be integrated into functioning end-to-end solutions.

This performance ceiling raises questions about agent readiness for production adoption in enterprise ML engineering, where partial solutions typically cannot replace human oversight.

Why the Disconnect Matters

The MLOps-Bench findings highlight a persistent challenge in agent evaluation. Standard benchmarks often measure whether an agent can pass a single test or generate correct code snippets. Production ML engineering, by contrast, demands orchestration across multiple systems, error handling in unfamiliar codebases, and decision-making under uncertainty—exactly the scenarios the benchmark isolates.

The 31.2% completion rate reflects real-world complexity: agents must navigate version control, dependency management, configuration files, and domain-specific patterns unique to each repository. A test-passing rate of 68.7% may indicate strong performance on isolated subtasks, but the inability to chain those subtasks into complete solutions reveals a fundamental limitation.

Open Access for Verification

The research team published both the benchmark dataset and source code publicly. The MLOps-Bench repository includes task definitions, metadata, and evaluation results. The underlying dataset documents the 31 source repositories and their metadata, enabling independent reproduction and future benchmarking efforts.

This transparency positions MLOps-Bench as a reference standard for evaluating agent progress on production-aligned tasks, offering the AI research community a measurable baseline for improvement.