Ranvier
An open-source LLM inference router that cuts time-to-first-token by 33–44% by sending each request to the backend GPU already holding the relevant KV cache, measured on 8× A100s across Llama-3.1 8B, 13B, and 70B. An adaptive radix tree over token prefixes gives O(L) lookups independent of key count, and sub-50µs routing decisions. Built shared-nothing and thread-per-core on Seastar — no locks on the hot path, string_view parsing over a single network buffer copy — with circuit-breaking health checks, gossip-based clustering, and Kubernetes-native deployment.
- 33–44%faster TTFT
- <50µsrouting decision
- 8× A100measured on