KV Cache

The per-token attention key/value tensors an LLM caches during decoding, which dominate inference memory and grow linearly with sequence length and batch size.

grounded in: Established inference technology directly underpinning the existing 'prefix-caching' and 'continuous-batching' nodes; the 'Distributed/decentralized LLM compute' trend theme (HN front page, 2026-07-12

Connected concepts

Prefix/KV-cache reuse, Continuous batching, Memory-bandwidth-bound decode, Model offloading, Distributed inference