Paper

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

Shows RL, unlike SFT, yields sparse near-orthogonal task updates in multi-task LLM training, then uses this to decouple tasks with Parallel-RL.

arXiv ↗Aug 6, 2026Code ↗

llm
reinforcement learning
multi-task learning
reasoning

Conceptual diagram illustrating SFT conflicts vs RL coexistence in policy space—core claim

What it is

Analyzes why multi-task SFT causes task interference while RL remains stable in LLM reasoning, and proposes Parallel-RL to exploit RL's near-orthogonal task updates. Instead of mixing tasks to avoid forgetting, it shows RL updates across tasks are sparse and near-orthogonal while SFT updates conflict; this motivates Parallel-RL, which trains tasks independently and merges RL updates.

Key results

Multi-stage RL gains 24.9% avg vs base; multi-stage SFT drops 23.1%
Single-task RL improves untrained tasks by 2.3% avg; SFT hurts them by 5.1%
RL task-update cosine similarity is ~1e-5 vs ~1e-1 to 1.0 for SFT

On-policy RL with advantage normalization naturally reduces cross-task gradient interference, making sequential multi-task training far more stable than SFT.

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

What it is

Key results

One useful ML email a week.

More from Aug 10 – Aug 16

Worth your time, not all of your time.