What it is
An 8B masked diffusion LLM trained from scratch with fully bidirectional attention, plus improved SFT and evaluation recipes. It markedly strengthens diffusion LM performance and is competitive with Qwen2.5 7B on several base benchmarks. Instead of switching to autoregressive training, it keeps masked diffusion end-to-end at 8B scale, pairing bidirectional pretraining with diffusion-style SFT, variable-length generation, and confidence-based MC scoring.
Key results
- Base avg 63.9 vs LLaDA's 51.1 and Qwen2.5 7B's 63.3
- Instruct MATH 56.7 vs LLaDA 42.2 and Dream 39.2
- Confidence scoring beats likelihood on HellaSwag 76.6 vs 74.3
Shows that scaled bidirectional diffusion training from scratch can match strong 7B autoregressive base models on several reasoning tasks.