ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration
IBM Research introduced ScarfBench, an open benchmark for evaluating AI coding agents on migrations among Spring, Jakarta EE, and Quarkus. The benchmark targets a practical weakness in software evaluations: code that compiles can still fail to deploy or behave incorrectly after a framework migration.
What is measured
The June 2026 benchmark paper describes 34 applications, 102 framework implementations, 204 migration tasks, about 151,000 lines of code, and 1,331 expert-written tests. A successful migration must build, deploy, and pass behavioral checks. That layered evaluation matters because build success alone can overstate whether an application is usable.
IBM Research reports that the strongest agents in its evaluation achieved less than 10% behavioral success, with results varying by migration direction and whole-application tasks proving especially difficult. The authors also report gaps between agents’ self-reported build results and independent verification. These are results from the benchmark’s evaluated agents and tasks, not a universal success rate for every coding agent or enterprise project.
Why teams should care
Framework migrations involve dependencies across configuration, persistence, services, and runtime behavior. ScarfBench offers researchers and engineering teams a way to test those capabilities beyond code translation. Before relying on an agent for a real migration, teams still need independent builds, deployment checks, and application-level tests.
Source: IBM Research’s ScarfBench overview · Benchmark leaderboard.