Repo

HarnessEval-W

Scores world-model rollouts with routed evaluation skills and saves an auditable case card with questions, answers, scores, and supporting frames.

GitHub ↗Aug 17, 2026Paper ↗

  • multimodal
  • world-models
  • benchmark
  • video-evaluation
Pipeline diagram showing skill routing, sub-agent reasoning, and final scoring method
Pipeline diagram showing skill routing, sub-agent reasoning, and final scoring method

What it is

Agent-based benchmark for world-model video evaluation that produces auditable reasoning traces instead of only scalar scores. Evaluates generated world-model videos with case-specific skill routing, sub-agent checks, and auditable evidence trees that justify each case score.

Key results

  • Writes summary.json, CSV/JSON leaderboards, and LEADERBOARD.md
  • Saves full case traces under model outputs and metric_cache
  • Routes by case context only, not by the model being evaluated

Shows that world-model benchmarking can be automated with inspectable reasoning rather than opaque aggregate metrics.

One useful ML email a week.

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.