Luca BertonThe numbers are staggering. The global AI inference market is projected to reach $255 billion by 2030. Startups are raising at valuations that would have been unthinkable two years ago: Baseten at $5B, Fireworks AI at $4B, Modal at $2.5B. And the fundamental shift is clear — 67% of all AI compute is moving from training to inference.
This isn't a bubble. It's the natural consequence of every enterprise discovering that training a model is the easy part. Serving it reliably, at scale, within budget, under regulatory constraints — that's where the real engineering happens. And increasingly, it happens on Kubernetes.
Here's why Kubernetes is winning the inference war, and what the four technical pillars of modern inference infrastructure look like.
The Market: Why Inference Is Eating AI
Training gets the headlines. Inference gets the revenue. Every ChatGPT query, every copilot suggestion, every fraud detection call, every medical imaging analysis — that's inference. And it runs 24/7.
The economics are brutal. A single large language model serving millions of users can cost $10-50M annually in compute alone. Multiply that across every enterprise deploying AI, and you understand the $255B projection.
| Company | Valuation | Focus |
|---|---|---|
| Baseten | $5B | Model inference platform, custom silicon |
| Fireworks AI | $4B | Compound AI systems, multi-model serving |
| Modal | $2.5B | Serverless GPU compute, developer-first |
| Together AI | $3.3B | Open-source model hosting, custom training |
| Cerebras | $4.25B | Wafer-scale inference, fastest token generation |
The Berkeley lab story is particularly telling. A single university research group spawned two billion-dollar inference companies — Anyscale (the Ray team) and Fireworks AI (the vLLM team). When academic researchers can build inference companies worth billions, the market signal is clear.
Pillar 1: Disaggregated (xPyD) Serving
Traditional LLM serving treats each request as a monolithic operation: one GPU handles prefill (processing the prompt) and decode (generating tokens) together. This is simple but wildly inefficient.
Prefill is compute-bound — it processes all input tokens in parallel. Decode is memory-bound — it generates one token at a time, limited by memory bandwidth. Forcing both onto the same GPU means you're always bottlenecked by whichever phase is running.
- Prefill + decode on same GPU
- GPU utilisation 30-40%
- Can't scale phases independently
- Hardware mismatch for both workloads
- Separate prefill and decode pools
- GPU utilisation 70-85%
- Scale each phase to demand
- Right hardware for each workload
The xPyD notation (e.g., 1P4D = 1 prefill node, 4 decode nodes) makes this concrete. For a chatbot handling long system prompts but generating short replies, you might run 1P8D. For a code completion service with short prompts but long outputs, 4P2D makes more sense.
This is exactly what llm-d brings to Kubernetes — production-grade disaggregated inference with automatic KV-cache transfer between phases. The result: 50-60% cost reduction at equivalent latency.
Hardware Profiling Per Phase
Disaggregation unlocks hardware specialisation. Prefill nodes want high FLOPS — NVIDIA H100 SXM, AMD MI300X, or even TPUs. Decode nodes want high memory bandwidth — H200 with its 141GB HBM3e, or upcoming B200 GPUs with 8TB/s bandwidth.
Some teams are even running prefill on GPU and decode on specialised ASIC accelerators. The economic arbitrage is significant: the right hardware for each phase can cut per-token costs by another 30% on top of disaggregation savings.
Kubernetes Recipes
Practical guide for container orchestration and deployment — hands-on patterns you can use today.
View on Amazon →Pillar 2: Kubernetes-Native Inference
Every major inference startup, every cloud provider, every enterprise AI platform is converging on Kubernetes. Not because it's trendy — because inference has the same orchestration problems that Kubernetes already solved for microservices.
Model serving needs:
- Auto-scaling — scale replicas based on queue depth, latency percentiles, or GPU utilisation
- Rolling updates — deploy new model versions without dropping requests
- Health checking — detect and replace unhealthy inference workers
- Resource isolation — prevent noisy neighbours from destroying latency SLAs
- Service discovery — route requests to the right model version on the right hardware
Kubernetes does all of this. The missing piece was GPU-aware scheduling and inference-specific routing, which is exactly what the Gateway API Inference Extension (GAIE) provides.
Gateway API Inference Extension (GAIE)
Announced at KubeCon EU 2026, GAIE introduces two new CRDs that make inference a first-class Kubernetes concept:
| CRD | Purpose | Key Fields |
|---|---|---|
| InferencePool | Groups GPU nodes serving the same model | selector, targetPort, extensionRef |
| InferenceModel | Maps model names to pools with traffic policy | modelName, criticality, weight, poolRef |
The criticality field is the game-changer. Set a model to "Critical" and Kubernetes guarantees capacity — it will shed "BestEffort" workloads before touching your production model. This is how you run cost-efficient batch inference alongside latency-sensitive real-time serving on the same cluster.
KV-Cache-Aware Routing
Standard load balancing (round-robin, least-connections) is terrible for LLM inference. Why? Because LLMs maintain a KV-cache — the accumulated context of the conversation. Route a follow-up request to a different GPU and that cache is gone, forcing a full re-computation of the prompt.
GAIE-compatible routers like llm-d's endpoint picker solve this with prefix-aware routing. They track which GPU has which conversation's KV-cache and route accordingly. The result: 40-60% latency reduction on multi-turn conversations.
Pillar 3: Tiered Prefix Caching
Every LLM-powered application shares common prefixes. System prompts, few-shot examples, retrieval context — the same text appears in thousands of requests. Without caching, every request recomputes these shared prefixes from scratch.
Tiered prefix caching creates a memory hierarchy specifically for KV-caches:
The savings are dramatic. For applications with shared system prompts (customer service bots, coding assistants, search), tiered prefix caching delivers 80-90% compute savings on the prefill phase. That directly translates to either lower cost or higher throughput at the same cost.
Why This Matters for Regulated Enterprises
Regulated enterprises typically standardise on a small number of approved models with extensive system prompts (compliance instructions, data handling rules, audit requirements). These prompts can be 10,000+ tokens — and they're identical across every request.
Without prefix caching, a bank processing 100,000 customer queries per day recomputes the same 10,000-token compliance prompt 100,000 times. With tiered caching, it computes it once and serves it from GPU memory for every subsequent request. The cost difference is measured in millions per year.
Evaluating RAG Solutions
Choose the right RAG model, configure, test, and optimise.
Start on Pluralsight →Pillar 4: Multi-Accelerator Support
The GPU monopoly is ending. NVIDIA still dominates, but the inference landscape now includes serious alternatives:
| Accelerator | Strength | Best For | Cost Index |
|---|---|---|---|
| NVIDIA H100/H200 | Ecosystem, software maturity | General inference, large models | 1.0x (baseline) |
| AMD MI300X | 192GB HBM3, memory capacity | Large context, memory-bound decode | 0.6-0.7x |
| Google TPU v5e | Cost per token, GKE integration | High-throughput batch inference | 0.4-0.5x |
| Intel Gaudi 3 | Price/performance, availability | Cost-sensitive mid-size models | 0.3-0.5x |
| AWS Inferentia2 | AWS-native, Neuron SDK | AWS-deployed standardised models | 0.3-0.4x |
The cost arbitrage is enormous. Running decode on AMD MI300X instead of NVIDIA H100 can save 30-40% at equivalent throughput for memory-bound workloads. Running batch inference on TPU v5e can save 50-60% versus NVIDIA.
But only if your infrastructure supports it. And this is where Kubernetes wins again.
The NVIDIA GPU Operator and Beyond
The NVIDIA GPU Operator proved the model: automated driver management, device plugin configuration, monitoring, all through Kubernetes operators. Now the same pattern is extending to every accelerator:
- AMD GPU Operator — ROCm driver management, MI300X device plugins
- Intel Device Plugins — Gaudi accelerator support via Kubernetes device plugins
- Google TPU Provisioner — GKE-native TPU pod scheduling with topology awareness
A single Kubernetes cluster can now schedule workloads across NVIDIA, AMD, and Intel accelerators simultaneously. Your inference router can make real-time decisions: route this latency-sensitive request to H100, route that batch job to MI300X, route the cost-insensitive summarisation to Gaudi.
Cost Arbitrage in Practice
One enterprise we advised runs their customer-facing chatbot on H100 (latency-critical, 200ms P99 SLA) and their internal document summarisation on MI300X (cost-optimised, 2s SLA acceptable). Same Kubernetes cluster, same deployment tooling, 40% lower total inference cost.
The key insight: not every inference request has the same SLA. Multi-accelerator support lets you match hardware cost to business criticality.
Why Kubernetes Wins
The four pillars converge on Kubernetes because each one requires the same thing: orchestration-aware infrastructure.
Needs pod-level scheduling of prefill vs decode pools, inter-pod KV-cache transfer, independent scaling per phase. Kubernetes pod topology, node affinity, and HPA handle this natively.
InferencePool and InferenceModel make models declarative resources. GitOps, RBAC, admission webhooks, audit logging — all the Kubernetes security and governance tools apply automatically.
Tiered caching needs topology-aware scheduling (which GPU has the cache?), persistent volume claims for NVMe tiers, and RDMA-capable network policies. Kubernetes provides the abstractions.
Device plugins, resource requests, node selectors, and scheduler extenders let you mix GPU types in one cluster. No other orchestrator offers this level of heterogeneous hardware support.
The alternative — building custom infrastructure for each pillar — is what every startup that didn't use Kubernetes eventually regretted. The ones that succeeded (vLLM, TensorRT-LLM, SGLang) all converged on Kubernetes as the deployment target.
EU AI Act Compliance Checklist
40-point checklist covering risk classification, data governance, transparency, and human oversight. Based on the official regulation.
Get Free Checklist →The AI Conformance Signal
At KubeCon EU 2026, CNCF announced Kubernetes AI Conformance (KARs) — a certification programme that guarantees GPU workloads are portable across any conformant Kubernetes distribution. This is the moment Kubernetes transitions from "good enough for AI" to "the standard for AI".
For regulated enterprises, conformance means vendor independence by specification. Your inference workloads aren't locked to a specific cloud provider's Kubernetes flavour. If a regulator requires you to move from AWS to a sovereign cloud, your InferencePool and InferenceModel manifests work unchanged.
Building Your Inference Strategy
If you're an enterprise planning inference infrastructure, here's the decision framework:
The Bottom Line
The inference gold rush is real. $255 billion in market value, billions in startup funding, and a fundamental shift in where AI compute happens. The winners will be the organisations that treat inference as an infrastructure discipline, not just a model deployment step.
Kubernetes isn't winning by default. It's winning because disaggregated serving, tiered caching, multi-accelerator scheduling, and inference-native CRDs all require the kind of orchestration that only Kubernetes provides at scale.
The question isn't whether to build on Kubernetes. It's how fast you can get the four pillars in place before your inference costs eat your AI budget.
Let's architect your inference infrastructure — we'll map your workloads to the right hardware, implement disaggregated serving, and build the Kubernetes-native platform that scales with your AI ambitions.