Paper

Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

Maohua Li, Qirui Li, Yanke Zhou, Yiduo Li, Zhaosheng Chi, Chao Xu, Cuifeng Shen, Yixuan Xu

A causal interpretability study shows structural chat-template tokens, not prompt tokens, maintain object identity in DiTs and yields a pruning rule.

arXiv ↗Jul 21, 2026Code ↗

multimodal
diffusion transformers
interpretability
attention pruning

Framework overview with four labeled analysis/method components explaining the causal interpretability approach

What it is

A causal analysis of text-to-image DiTs finds chat-template tokens, not prompt tokens, become the main semantic carriers during denoising and enable pruning. Instead of treating chat-template tokens as inert formatting, the paper shows they become semantic registers inside DiTs: image latents write object identity into them, while prompt-reading heads are largely dispensable and can be pruned.

Key results

Pruning prompt-attending heads cuts 20% attention FLOPs for only -1.4 GenEval
On GenEval, template tokens get 76% of I2T attention in Qwen-Image
Per-token template attention is 6.4x prompt-token attention on GenEval

In joint-attention DiTs, the tokens that encode semantics at input need not be the tokens that preserve semantics during generation.

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

What it is

Key results

One useful ML email a week.

More from Jul 20 – Jul 26

Worth your time, not all of your time.