Speculative decoding

An inference acceleration method where a small draft model proposes tokens that the larger target model verifies in parallel, cutting latency without changing outputs.

grounded in: HN trend theme 'Local LLM inference on modest hardware: builders squeezing frontier models onto slow/small machines', a durable technique supported by the system's vLLM serving stack.

Connected concepts

vLLM / PagedAttention, Test-time compute, Local-first inference, Quantization, Memory-bandwidth-bound decode, Small language model