Singularity-Bench Finds Opus 5.5 Agent Matches Humans
A research team introduced Singularity-Bench, an open-ended benchmark designed to test whether AI agents can autonomously build AI systems end-to-end, according to a paper posted this week (alphaXiv). The benchmark covers six distinct tasks spanning the full AI-development pipeline: pre-training, mid-training, post-training, inference, agent harness construction, and cluster orchestration (alphaXiv).
The Benchmark
The evaluation set included 11 agents built from seven different frontier models paired with two agent harnesses, according to the paper and an accompanying writeup (Pub-AI). Each agent's performance was scored using what the authors call the Singularity Index, defined as the mean of an agent's score relative to a human-engineered baseline across all six tasks (alphaXiv). A score of 1.0 would indicate parity with human engineering teams on average; scores above 1.0 indicate the agent outperformed the human reference.
The Results
One configuration stood out sharply from the rest: Opus 5.5 running inside the Claude Code harness posted a Singularity Index of 1.252, meaning it matched or exceeded the human-engineered reference system on all six tasks (Benchmark Radar). The remaining ten agents in the test pool averaged just 0.58 on the same index, a gap of more than double (GitHub).
The margins were not uniform across tasks, however. In post-training specifically, the Opus 5.5/Claude Code combination scored 0.533 against a human reference score of 0.539 — a difference the authors describe as statistically comparable rather than a clear win (alphaXiv). That qualification suggests the headline 1.252 figure is driven by outsized performance on some of the other five tasks, even as the post-training result shows the agent essentially tied with human engineers rather than surpassing them.
Why It Matters
Singularity-Bench arrives as labs and independent researchers increasingly probe whether AI systems can take over the engineering work of building other AI systems — not just writing code snippets, but handling orchestration, training pipelines, and inference infrastructure as a connected whole (zot.news). The wide spread between the top-performing configuration and the ten-agent average also underscores how much current results depend on the specific model-and-harness pairing, rather than being a generic property of "frontier models" as a category.


