Papers.Data.Code
Paper

Language Models Can Control Their Own Attention

Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

A prompting-plus-runtime scheme where the model declares which context region to attend, and the engine masks KV blocks without any auxiliary scorer.

arXiv ↗Sep 2, 2026

  • nlp
  • llm
  • long-context
  • efficient inference
  • sparse attention
Illustrates Declarative Attention method with modes, token reduction, and attention mask diagram
Illustrates Declarative Attention method with modes, token reduction, and attention mask diagram

What it is

Lets an LLM declare its own attention scope during reasoning, so inference can skip most KV-cache reads on long-context decoding without retraining. Instead of proxy scorers that still scan context every step, the model emits parseable global/focus/local attention declarations in CoT; the runtime masks KV blocks accordingly on off-the-shelf LLMs.

Key results

  • 52.0% fewer attended decode tokens on Gemma-4-31B with 1.27pp drop
  • 31.1% fewer attended decode tokens on Qwen-3.6-27B with 2.75pp drop
  • Up to 21M attended-token savings per response as context length grows

Shows that off-the-shelf LLMs can externalize enough of their own retrieval plan to drive useful sparse attention at decode time.

One useful ML email a week.

More from Sep 07 – Sep 13

The whole week →

Picked Sep 11

Picked Sep 10

Worth your time, not all of your time.

We track new ML work across papers, code, and data, then keep only the pieces worth a closer look. Selected, not collected.

Papers.Data.Code

The Weekly ML Signal

Selected, not collected.

One useful ML email a week: the papers, repos and datasets worth your attention.

Worth your time, not all of your time.