raghu@dark-factory :~/kb/cache-aware-routing $ cat

KV-cache-aware routing

Load-balancing that steers a request to the inference replica already holding the largest matching prefix in its KV cache, maximizing prefix-cache hits instead of routing blindly.

grounded in: Established production LLM-serving technique layered on prefix caching; extends the system's real tiered router (`router_stats.py`, `tiered-router`) and vLLM prefix-cache stack.

Connected concepts

Prefix/KV-cache reuse, Tiered LLM router, Prefill/Decode Disaggregation, KV Cache, Backpressure

Explore it live in the knowledge graph →