Sparse attention

An attention technique that computes only a selected subset of query-key interactions instead of the full quadratic matrix, letting models train and serve million-token contexts far more cheaply.

grounded in: Latest-trends NOTABLE: 'Flash-MSA: Sparse-attention kernels that accelerate million-token training runs.'

Connected concepts

KV Cache, Memory-bandwidth-bound decode, PagedAttention, Chunked prefill, Mixture-of-Experts