What it is
Append-only benchmark that tracks whether a shipped frontier model drifts in capability over time using pinned prompts, exact graders, and pre-registered stats. Measure whether a deployed Claude model gets worse after launch by running a locked daily panel through a pinned Claude Code harness, logging raw transcripts, and comparing 10-day windows to launch-week baseline.
Key results
- Uses a 78-item panel selected from 2,336 screened questions
- Detects ~7.5-point accuracy shifts per 10-day window at planned budget
- Validation: effort low caused -62% output tokens and -8.3±4.5 accuracy pts
Shows that post-launch model drift can be monitored credibly with a fixed harness, append-only logs, and pre-registered paired evals.