What it is
Structured sparse attention splits long-range causal history across KV heads while keeping full coverage collectively. It preserves model quality near full attention with much lower long-context cost. Instead of giving every head the full prefix, CoWA shares local and sink windows across heads and partitions distant history into complementary head-specific windows. This keeps full causal coverage collectively, with a fixed position-based pattern and no router.
Key results
- 128K: 7.4x/8.6x lower train fwd/bwd latency vs FullAttn
- 128K decoding: 3.0x lower latency and 7.6x lower peak memory vs FullAttn
- 8K recall: 89.73% vs 89.97% FullAttn; DSA 53.71%, MoBA 50.12%
Full causal access need not be duplicated per head; distributing long-range context across heads preserves quality while cutting attention cost.