ScarfBench: A Real-World Test for Enterprise Java Migration AI

Enterprise Java Modernization Remains a Weak Point for AI Agents

According to Hugging Face, IBM Research has released ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark designed to evaluate AI agents on Java framework migration tasks. The benchmark comprises 34 real applications, 102 framework implementations, and 204 migration tasks totaling approximately 151,000 lines of code. Unlike traditional software engineering benchmarks that measure code-generation accuracy against reference implementations, ScarfBench enforces a stricter requirement: migrated applications must build successfully, deploy correctly, and pass behavioral validation tests.

This distinction reflects a fundamental gap in how current AI benchmarks measure progress. State-of-the-art coding agents perform well on tasks like bug fixing and isolated code generation, yet struggle with the interdependent challenges of framework migration—translating not just syntax, but the underlying architectural assumptions baked into dependency injection, persistence configuration, and build system descriptors across three major Java ecosystems: Spring, Jakarta EE, and Quarkus.

Why Java Migration Remains Difficult

Framework migration extends far beyond simple annotation replacement. According to Hugging Face’s analysis, a single repository shift—say, from Spring to Jakarta—can cascade into changes across multiple layers: dependency injection containers, ORM query syntax, persistence metadata, and XML or annotation-based configuration files. A misconfigured field in any of these components prevents deployment, making the task one of semantic translation rather than lexical substitution.

The benchmark includes 1,331 expert-written validation tests and both focused migration tasks and whole-application migrations. The latter—moving an entire codebase across frameworks—presents particular difficulty for current agents, whose performance drops significantly when context expands beyond a single module.

Current Agent Performance Reveals the Scale of the Challenge

Hugging Face reports that even frontier agents achieve less than 10% behavioral success on ScarfBench, with success rates varying considerably by framework pairing. This low ceiling suggests a meaningful gap between capabilities on synthetic benchmarks and requirements for production-grade modernization work. The benchmark’s emphasis on behavioral correctness—rather than syntactic similarity or pass-rate metrics—likely accounts for the performance drop relative to agents’ results on datasets like HumanEval or SWE-bench.

Why This Matters

Enterprise application modernization is a billion-dollar annual activity for large organizations seeking to improve cloud readiness, maintainability, and developer productivity. If AI agents cannot reliably handle framework migration at scale, enterprises will continue to rely on manual refactoring or targeted tooling, limiting the productivity gains from automation.

For AI developers, ScarfBench identifies a concrete class of tasks where current agent architectures—typically built around code generation, not semantic system translation—reach their limits. The benchmark’s open release creates both an evaluation venue and a target for specialized agents trained on the semantic dependencies of Java frameworks. Success here would unlock a significant economic use case; the current 10% ceiling suggests this problem remains unsolved.