Linear attention

Attention variants that replace the quadratic softmax with kernel/feature maps or recurrent state to reach near-linear cost in sequence length for cheaper long-context serving.

grounded in: Trend theme 'LLM interpretability and efficient training ... cheaper long-context kernels' — a durable long-context efficiency paradigm distinct from the softmax kernels already in the graph (flash-/r

Connected concepts

Sparse attention, FlashAttention (IO-aware attention), KV Cache, Memory-bandwidth-bound decode