What it is
Trains language models with built-in concept bottlenecks and attribution interfaces, arguing interpretability can scale with capability instead of hurting it. Instead of explaining opaque LMs post hoc, it trains interpretability into the model: concept bottlenecks, trained absence baselines, and training-data attribution make outputs traceable and steerable without retraining.
Key results
- Within ~10% of open peers' avg benchmark score with 2-16x less compute
- Validation loss predicted within 0.11 nats from small-scale scaling fits
- 3 of 4 interpretability metrics were predicted within tight scaling bounds
Interpretability constraints need not trade off with scale; they can remain competitive while making representations more human-aligned.