OS-Bench: AI agents fail 70% of cross-OS code migrations
# OS-Bench Tests AI Agents on Cross-Operating-System Code Migration
Researchers published OS-Bench, a new evaluation framework for testing software engineering agents on their ability to migrate codebases between operating systems, finding that even the best-performing configurations struggled to complete the task reliably.
The benchmark, released October 14, 2026, comprises 140 validated repository-level migration tasks drawn from 111 repositories spanning five programming languages: C, C++, C#, Go, and Rust arxiv.org. The task set includes 91 Windows-to-Linux migrations and 49 Linux-to-Windows migrations, testing whether agents can successfully port codebases across platform boundaries while preserving correctness on both operating systems.
Evaluation Methodology and Scale
The research team evaluated six different models using four agent frameworks: SWE-agent, Claude Code, OpenHands, and Win-agent. The scope reflects real-world infrastructure challenges: code migration between major operating systems is a routine but labor-intensive task that AI agents are increasingly expected to automate.
Performance Results Show Significant Gaps
The best configurations achieved 26.4% success on Linux-target tasks and 20.4% on Windows-target tasks while maintaining correctness validation on both platforms arxiv.org. These results highlight a substantial capability gap in agent reasoning about platform-specific dependencies, system calls, build configurations, and API variations that distinguish Windows from Linux environments.
The 70% failure rate across both directions signals that current agent architectures lack robust mechanisms for handling cross-platform concerns—a critical limitation for enterprises managing heterogeneous infrastructure stacks. The asymmetry between Linux-target and Windows-target success rates suggests agents may have stronger performance when migrating to POSIX-compliant systems, though both rates remain well below production-grade reliability thresholds.
Implications for Agent Deployment
The benchmark establishes a concrete, reproducible evaluation standard at a moment when enterprises are piloting autonomous agents for DevOps and infrastructure automation. OS-Bench provides a structured way to measure progress on an economically significant problem: reducing manual effort in maintaining codebases across platform variants.
The research also signals where future agent development efforts should concentrate—platform abstraction layers, environment detection, and systematic testing of system-level APIs appear to be critical blind spots in current agent reasoning.
Next Steps
The full benchmark and evaluation details are available on arXiv, enabling other research teams and vendors to test their agent implementations against the same task set and potentially contribute improved configurations.
