FlashAttention (IO-aware attention)

An IO-aware exact-attention algorithm that tiles the computation and avoids materializing the full attention matrix in high-bandwidth memory, cutting memory traffic to accelerate long-context training and inference.

grounded in: Notable trend item 'Flash-MSA: Sparse-attention kernels that accelerate million-token training runs' — an evolution of IO-aware FlashAttention-style kernels.

Connected concepts

Kernel fusion, Memory-bandwidth-bound decode, Sparse attention, KV Cache