llm-d 1.0 ships as first stable release of open Kubernetes inference stack

llm-d 1.0 is the first stable release of an open-source, Kubernetes-native distributed inference stack. It turns vLLM or SGLang model servers into one coordinated pool. The release was announced on October 20, 2026, at the PyTorch Conference North America, and it targets self-hosted agent and RL workloads, where round-robin load balancing breaks down.

The project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Red Hat, IBM and Google are the lead maintainers. It counts more than 500 contributors from over 25 organizations, is licensed under Apache 2.0, and became a CNCF Sandbox project on March 12, 2026. It runs on Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs.

Techniques like prefix-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference already shipped before 1.0. This release is about running them reliably in production, which IBM calls day 2 operations. New controls include multi-tenant quality of service with priority handling and zero-downtime updates for shared inference.

On performance, Google reports a median time to first token cut by more than 85%, with 2.4x throughput, on an Ironwood TPU coding workload. It also points to a published benchmark on a 17,000-node multi-region GKE deployment. These are vendor-reported numbers. The weakest result is for RL: time-slicing lifts accelerator duty cycle only to above 70%, leaving roughly 30% idle time.

Our take

A stable 1.0 backed by Red Hat, IBM and Google gives teams a credible open alternative to stitching together their own routing and cache logic. The headline gains are vendor-reported on one TPU workload, so treat them as a ceiling, not a promise.