llm-d: How Kubernetes-Native Distributed Inference Cuts LLM Serving Costs 60%

LLM inference is expensive. At scale — millions of requests per day for customer service, document processing, code generation — GPU compute for inference often exceeds training costs. The standard approach: deploy vLLM or TGI on GPU nodes, scale replicas, and accept the bill.

The problem with this approach is that LLM inference has two distinct computational phases with very different hardware requirements, and running both on the same hardware wastes money:

• Prefill phase: Processing the input prompt. Compute-intensive (matrix multiplications over the entire input). Benefits from high FLOPS (A100, H100). Duration scales linearly with input length
• Decode phase: Generating output tokens one at a time. Memory-bandwidth intensive (reading KV cache for each token). Benefits from high memory bandwidth, but doesn't need peak FLOPS. Duration scales with output length

When you run both phases on an A100 (optimised for compute), the decode phase wastes 60-70% of the GPU's compute capability — it's bandwidth-bound, not compute-bound. You're paying for FLOPS you don't use.

llm-d — created by Brian Stevens (SVP, AI CTO at Red Hat) and announced as a CNCF project at KubeCon EU 2026 — solves this with disaggregated serving: route prefill to compute-optimised hardware and decode to bandwidth-optimised hardware. Same quality. Lower cost.

The Architecture: xPyD Disaggregated Serving

llm-d introduces the xPyD model — x prefill nodes, y decode nodes — where x and y scale independently based on workload characteristics:

The llm-d architecture routes requests through a smart router that separates prefill (compute-heavy) from decode (bandwidth-heavy), transferring the KV cache between phases:

KV cache transfer via shared memory / RDMA

The cost difference at scale (100k requests/day, avg 2k input + 500 output tokens):

Approach	Hardware	Monthly Cost	Savings
Monolithic	4x A100	8,000 EUR	—
llm-d disaggregated	2x A100 + 4x L4	6,000 EUR	25%
llm-d + prefix caching	1x A100 + 4x L4	4,000 EUR	50%

Tiered Prefix Caching

Enterprise LLM workloads have a pattern that llm-d exploits: shared prefixes. Consider a customer service agent:

A typical enterprise agent request is built from four layers, each with different sharing characteristics:

Layer	Shared Across	~Tokens
System prompt "You are a customer service agent for Acme Corp..."	All requests	2,000
RAG context "Product documentation for Widget X..."	Similar queries	3,000
Conversation history "Customer: I need help with..."	Per session	1,000
Current query "What's the return policy?"	Unique	100

The caching impact is dramatic:

Scenario	Tokens Processed	Reduction
No caching	6,100	—
System prompt cached	4,100	33%
System + RAG cached	1,100	82%

llm-d implements tiered prefix caching:

• L1 — GPU memory cache: Hot prefixes stored in GPU HBM. Sub-millisecond access. Limited by GPU memory capacity
• L2 — Host memory cache: Warm prefixes in CPU RAM. ~1ms access. Much larger capacity (hundreds of GB)
• L3 — Distributed cache: Cold prefixes shared across nodes via RDMA or NVMe. ~5ms access. Cluster-wide prefix sharing

For enterprise deployments where thousands of requests share the same system prompt and similar RAG contexts, tiered caching reduces prefill compute by 60-80%. That's 60-80% fewer A100 GPU-seconds per request — the savings are enormous at scale.

Kubernetes-Native: Why It Matters

llm-d isn't a standalone system — it's Kubernetes-native. This is the key differentiator from existing inference solutions:

• Scheduling via Kueue: Prefill and decode pods scheduled through Kueue with proper GPU affinity. Prefill pods request high-FLOPS GPUs; decode pods request high-bandwidth GPUs. The scheduler understands the difference
• Scaling via KEDA/HPA: Prefill and decode scale independently. Long prompt burst? Scale prefill. High concurrent users? Scale decode. Traditional auto-scaling scales both together — wasting resources on the phase that doesn't need it
• Observability via Prometheus: Per-phase metrics — prefill latency, decode throughput (tokens/sec), cache hit rates, KV transfer latency. Standard Grafana dashboards, standard alerting
• Policy via Kyverno: Which teams can deploy llm-d inference? Which models are approved? What are the GPU quotas? All enforced through standard Kubernetes policy — the same Kyverno policies that govern the rest of your platform
• GitOps via ArgoCD: llm-d configurations (model, hardware mapping, cache settings, scaling rules) defined in Git and deployed via ArgoCD. Change model? PR review → merge → auto-deploy. Same workflow as every other Kubernetes workload

Multi-Accelerator Support

llm-d isn't locked to NVIDIA. The disaggregated architecture naturally supports heterogeneous hardware:

llm-d isn't locked to NVIDIA. The disaggregated architecture naturally supports heterogeneous hardware:

The router selects accelerators based on a cost-optimised strategy — prefer the cheapest available hardware, with automatic fallback if the preferred type is unavailable.

For European sovereign cloud deployments, multi-accelerator support is critical. You might run prefill on NVIDIA A100s (best performance) but decode on Intel Gaudi or AMD MI300 (available from European neocloud providers). Same inference quality, broader hardware sourcing, reduced vendor dependency.

When to Use llm-d

llm-d isn't for every inference workload. Use it when:

• High volume: More than 10,000 inference requests per day. Below this, the operational complexity of disaggregated serving doesn't pay for itself
• Long prompts: Average input length above 1,000 tokens. Short prompts have minimal prefill cost — disaggregation saves less
• Shared prefixes: Enterprise agents with common system prompts and RAG context. If every request is completely unique, prefix caching provides no benefit
• Cost-sensitive: When inference cost is a material budget item (typically above 5,000 EUR/month). Below this, optimisation effort exceeds savings
• Latency-tolerant: The KV cache transfer between prefill and decode adds 5-20ms. For sub-50ms latency requirements (real-time trading), monolithic serving on high-end GPUs may still be preferred

Combining llm-d with the AI Platform Stack

llm-d is a serving layer. It integrates into the broader AI platform engineering stack:

• Model registry → llm-d: Models promoted from staging to production deploy to llm-d serving infrastructure. The model registry tracks which version is active on which llm-d cluster
• Agent frameworks → llm-d: LangChain, CrewAI, or custom agents call llm-d endpoints. The agent doesn't know inference is disaggregated — it sees a standard OpenAI-compatible API. Prefix caching is transparent
• Guardrails → llm-d: Input/output guardrails run as middleware before/after llm-d inference. PII filtering, content safety, cost limits — all enforced at the platform level
• Cost tracking → llm-d: Per-request cost attribution: prefill GPU-seconds + decode GPU-seconds + cache storage. Teams see exactly what their inference costs, broken down by phase

What Open Empower Delivers

• llm-d assessment: Analyse your inference workloads to determine if disaggregated serving will deliver meaningful savings. Workload profiling: prompt lengths, prefix commonality, volume, latency requirements
• Architecture design: Design the xPyD topology for your workloads — how many prefill nodes, how many decode nodes, which accelerators, what cache configuration
• Kubernetes deployment: Deploy llm-d on your Kubernetes infrastructure with Kueue scheduling, KEDA auto-scaling, Prometheus observability, and ArgoCD GitOps
• Prefix cache optimisation: Analyse your prompt patterns and configure tiered caching — L1/L2/L3 — for maximum cache hit rates and minimum prefill compute
• Multi-accelerator: Configure llm-d for heterogeneous hardware — NVIDIA, AMD, Intel — with cost-optimised routing and sovereign cloud compatibility
• Cost modelling: Before/after cost analysis comparing monolithic serving vs llm-d disaggregated. Per-request cost attribution for chargeback/showback

LLM inference doesn't have to be expensive. It has to be architected. Let's design your inference infrastructure.