What it is
Extracts hindsight skills from an agent's own on-policy trajectories and distills them into dense token-level RL signals for long-horizon language agents. Instead of relying on external skill memories or retrieved hints, it distills skills mined from completed on-policy trajectories. It routes step-level skills at critical decisions and episode-level skills elsewhere, then uses the policy's log-prob shift under skill-augmented context as token-level advantage.
Key results
- WebShop: 74.2 success vs 68.0 SDAR and 63.3 GRPO on Qwen2.5-3B
- ALFWorld: 90.0 avg vs 85.9 SDAR and 81.2 GRPO on Qwen2.5-7B
- Search QA: 49.2 avg vs 49.0 RLSD/SDAR and 42.0 GRPO on Qwen2.5-7B
Shows that distribution-matched hindsight skills from the current policy improve credit assignment in long-horizon agent RL without needing privileged inference-time context.