What it is
A looped MoE Transformer uses layer-wise recurrence plus a compute-matched scaling recipe to beat vanilla Transformers at the same training cost. Instead of repeating the whole stack like prior looped LLMs, it loops each layer locally and uses the saved activation memory to widen the model under the same wall-clock pretraining budget.
Key results
- 92.10% on AIME, above Qwen3-30B-A3B-Thinking at 90.10%
- 94.21% on AMC, above gpt-oss-20B-High at 91.05%
- 84.72% on IFEval, above Nemotron-3-Nano-30B-A3B at 77.05%
Shows looped transformers can outperform larger vanilla baselines when recurrence is paired with the right compute-matched width-depth tradeoff.