By Connie · Last reviewed: April 2026 — pricing & tools verified · This article contains affiliate links. We may earn a commission at no extra cost to you if you sign up through our links.

Google llm-d: The Open-Source Project Reshaping Distributed AI Inference

On April 2, 2026, five of the most influential names in enterprise infrastructure announced a joint initiative that could do for AI serving what Kubernetes did for container orchestration. llm-d — short for LLM Distributed — is now an official CNCF (Cloud Native Computing Foundation) Sandbox project, backed by Google Cloud, Red Hat, IBM Research, CoreWeave, and NVIDIA.

The timing is not coincidental. Frontier models have grown so large that running them on a single GPU is no longer viable. GPT-5.4 and Claude Opus 4.6 each require hundreds of accelerators to serve at production scale. The industry has fragmented around proprietary serving stacks — every cloud provider builds its own. llm-d is a bet that the market needs an open standard.

What llm-d Actually Does

llm-d is not a new language model. It is a distributed orchestration layer that sits above existing inference engines like vLLM, TGI (Text Generation Inference), and TensorRT-LLM. Think of it as the scheduling and routing brain that coordinates how requests flow across a fleet of GPU workers.

The core architectural insight behind llm-d is disaggregated prefill and decode. In standard LLM inference, every request goes through two phases: prefill (processing the input prompt) and decode (generating each output token one by one). These have very different compute profiles. Prefill is compute-heavy and latency-tolerant. Decode is memory-bandwidth-bound and latency-sensitive. llm-d routes them to different hardware pools — maximizing throughput without sacrificing response time.

The Five Founding Partners and Their Roles

Why CNCF Matters Here

Submitting llm-d to the CNCF Sandbox is a deliberate governance move. CNCF (the foundation behind Kubernetes, Prometheus, and Envoy) provides neutral stewardship — no single company controls the roadmap. This directly addresses the concern that enterprise AI infrastructure would permanently splinter along cloud provider lines.

Amazon Web Services, Microsoft Azure, and Oracle Cloud are not founding partners, which means llm-d is immediately positioned as a challenger to their proprietary serving stacks. AWS has SageMaker LMI (Large Model Inference). Azure has its own vCore-based serving layer. Whether those ecosystems adopt llm-d or resist it will define how this plays out over the next two years.

llm-d vs. Existing Serving Frameworks

The KV Cache Disaggregation Innovation

One of llm-d's most technically significant contributions is KV cache disaggregation — a mechanism developed in collaboration with IBM Research. In standard serving, the KV (key-value) cache generated during prefill lives on the same GPU that will run the decode phase. This creates imbalanced memory pressure and forces over-provisioning.

llm-d moves the KV cache to a shared memory tier that both prefill and decode workers can access. This is architecturally similar to what Google TurboQuant does at the compression layer — both approaches attack the same root problem: GPU memory is the binding constraint on LLM serving cost and throughput.

What This Means for Enterprise AI Buyers

For enterprises running private model deployments — whether on-premise or in their own cloud accounts — llm-d solves a real and expensive problem. Today, standing up a production serving cluster for a 70B+ parameter model requires deep expertise in CUDA, tensor parallelism, and cloud networking. Most organizations do not have that expertise.

llm-d's Kubernetes-native design means it fits into existing DevOps workflows. Helm charts, Prometheus metrics, and standard Kubernetes RBAC all work out of the box. For Red Hat OpenShift customers in particular, llm-d is expected to become the default AI serving layer in OpenShift AI by Q3 2026.

Frequently Asked Questions