What it is
Introduces Taste-Bench, a benchmark for long-horizon decision quality in agent trajectories, and shows taste can be improved by distilling hindsight judgments. Instead of scoring only end-to-end success, it isolates decision forks from agent trajectories and tests which branch a model would choose before outcomes are known; taste is then trained by distilling a hindsight teacher's judgment.
Key results
- Best Taste-Bench accuracy is 59.7% on 502 binary fork questions
- Mean accuracy drops from 62.3% to 21.0% as evidence moves later
- More reasoning changes accuracy by only -0.2 pts and +2.2 pts on 2 models
Long-horizon agent quality depends on a distinct decision-making skill that end-to-end benchmarks miss, but hindsight supervision can train it.