What it is
Targets the real deployment policy in LLM RL, not just the training engine policy, by accepting updates only when the synchronized inference policy is likely to improve. Instead of optimizing trainer-side policy updates and hoping deployment improves, it optimizes for monotonic inference-policy improvement via sampler-referenced updates plus inference-side accept/reject rollback.
Key results
- Best average reasoning performance on Qwen3-4B under FP8 rollout
- Best average reasoning performance on Qwen3-1.7B under FP8 rollout
- More stable training dynamics than prior mismatch-handling methods
Shows that under training-inference mismatch, stabilizing the trainer policy is not enough; update selection must target the deployed inference policy.