What it is
A benchmark for forward-deployed AI delivery where agents must clarify ambiguous customer needs, work in live-like systems, and improve business KPIs. Unlike well-specified agent benchmarks, it evaluates end-to-end delivery in simulated customer environments, scoring KPI uplift minus stakeholder-contact cost and testing whether agents can uncover unstated constraints.
Key results
- Best L3 score is 0.56 mean; only 14 of 95 tasks resolved
- Oracle per-task model choice still reaches only 0.70 and 30/95 resolved
- Top models drop from 0.73-0.82 to 0.39-0.62 when constraints must be found
Shows the main gap in enterprise agent deployment is uncovering unstated constraints and turning them into reliable KPI gains.