Long Context
DoSA: Dots Sparse Attention
Sparse attention reads few keys, but choosing them still means scoring every key with every indexer head. DoSA proves most keys cannot win and skips them, with exactly the same Top-K.
Sparse attention reads few keys, but choosing them still means scoring every key with every indexer head. DoSA proves most keys cannot win and skips them, with exactly the same Top-K.