DoSA: Dots Sparse Attention

Tue, 20 Oct 2026 00:00:00 +0800

Mixture-of-experts models made feed-forward layers conditional: each token pays for a few experts, not all of them. Sparse attention promises the same for attention, since each query reads only a small set of keys. In practice the promise stops short. DeepSeek Sparse Attention (DSA) picks those keys with a multi-head indexer, and the indexer scores every visible key with every indexer head before taking the Top- $K$ . The attention is sparse; the selection is dense.

About

Mon, 01 Jan 0001 00:00:00 +0000

llm-sys studies how large language model architectures evolve and how to make them efficient: attention and sparsity, model structure, and the kernels and serving stacks that run them. We publish papers with code and write up the ideas here.