KV-cache-aware routing
Load-balancing that steers a request to the inference replica already holding the largest matching prefix in its KV cache, maximizing prefix-cache hits instead of routing blindly.
grounded in: Established production LLM-serving technique layered on prefix caching; extends the system's real tiered router (`router_stats.py`, `tiered-router`) and vLLM prefix-cache stack.
Connected concepts
Prefix/KV-cache reuse, Tiered LLM router, Prefill/Decode Disaggregation, KV Cache, Backpressure
Explore it live in the knowledge graph →