What it is
An expert-curated benchmark for large, multilingual code refactoring tasks that fixes test/spec flaws in prior SWE benchmarks and remains unsaturated for current agents. Instead of mostly single-file bug-fix tasks with flawed tests, it benchmarks behavior-preserving refactors from real commits, with rewritten specs and manually cleaned tests across 7 languages.
Key results
- Best model reaches only 41.2% resolve rate on 170 tasks
- Tasks average 11.4 modified source files and 261.6 LOC
- 30% of tasks modify >10 files; 32% change >200 LOC
Current coding agents still struggle mainly with cross-file coordination on large, behavior-preserving refactors.