Efficient backbones for super intelligence
We study how model architectures change, from attention and sparsity to the kernels that run them, and how to get the same model quality for less compute. Posts here explain our papers and the code behind them.
DoSA: Dots Sparse Attention
Sparse attention reads few keys, but choosing them still means scoring every key with every indexer head. DoSA proves most keys cannot win and skips them, with exactly the same Top-K.
Open source
All projectsDoSA
DoSA in SGLang. Certified screening for the sparse-attention indexer, with exact Top-K selection.