What it is
Analyzes why on-policy distillation makes students over-generate: teacher and student often disagree on which EOS token means stop. It fixes this by supervising EOS tokens as one semantic stop action. Instead of only matching sampled surface tokens, it merges functionally equivalent EOS tokens into one semantic stop action. This targets a failure prior work missed: teacher and student can prefer different EOS tokens even with the same declared stop set.
Key results
- Across Qwen3, Llama 3.2, and Gemma 3, EOS aggregation mitigates length inflation
- On Qwen3, shared-stop decoding alone tracks vanilla OPD and fails to fix clipping
- In Qwen3, student prob. of teacher-preferred <|im_end|> is ~1e-11 near terminal states
Length inflation in OPD can come from tokenization-level disagreement about how to stop, not just broader optimization instability.