Efficient backbones for super intelligence
We study how model architectures change, from attention and sparsity to the kernels that run them, and how to get the same model quality for less compute. Posts here explain our papers and the code behind them.
DoSA++: One Query, One Scan for Sparse Attention
Sparse attention reads few keys, but choosing them can still require a full multi-head scan of the context. DoSA++ turns this search into one dot product per key. It estimates head activation rates from high-scoring samples, folds all query heads into a single direction, and uses the resulting scores to select candidates for refinement with the original multi-head indexer. The method adapts to each query and its current key cache without retraining. On GLM-5.2, DoSA++ accelerates time to first token by 2.35× at 1M tokens and the complete indexer by up to 8.01×. It recovers 99.96% of native DSA’s selected keys on RULER and 99.90% on LongBench v2, while maintaining RULER performance and improving LongBench v2 accuracy.
More posts
All postsDoSA: Dots Sparse Attention
Sparse attention reads few keys, but choosing them still means scoring every key with every indexer head. DoSA proves most keys cannot win and skips them, with exactly the same Top-K.
Open source
All projectsDoSA
DoSA in SGLang. Certified screening for the sparse-attention indexer, with exact Top-K selection.