Multi-Tenant GPU Orchestration on OpenShift AI: Lessons from Production

GPU infrastructure is expensive. A single NVIDIA A100 node costs $15,000–$30,000+ per month in cloud pricing. When multiple teams need GPU access — data scientists training models, ML engineers running inference, researchers experimenting — you can't give everyone their own cluster. You need multi-tenancy. And multi-tenant GPU orchestration is harder than it looks.

This post shares lessons from real production deployments on OpenShift AI, where we orchestrated GPU workloads across multiple teams while maintaining isolation, fairness, and cost efficiency. These are the insights behind my KubeCon EU 2026 talk.

The Multi-Tenancy Challenge

GPU multi-tenancy has three tensions that don't exist with CPU workloads:

• GPU indivisibility — You can't split a GPU like you split CPU cores (unless you use MIG or time-slicing, each with tradeoffs)
• Memory contention — GPU memory is fixed and non-swappable. An OOM kill on GPU memory is catastrophic for long-running training jobs
• Scheduling complexity — GPU topology matters. Two GPUs on the same NVLink interconnect perform vastly differently than two GPUs across PCIe

OpenShift AI GPU Architecture

OpenShift AI (formerly RHODS) provides a managed ML platform on top of OpenShift. For GPU orchestration, the stack is:

• NVIDIA GPU Operator — Manages GPU drivers, container toolkit, device plugin, and monitoring
• NVIDIA MIG Manager — Configures Multi-Instance GPU partitioning on A100/H100
• OpenShift Resource Quotas — Limits GPU allocation per namespace/team
• Kueue — Fair queuing and preemption for batch workloads
• Node Feature Discovery — Labels nodes with GPU capabilities for topology-aware scheduling

MIG vs Time-Slicing vs Whole GPU

The first decision: how to partition GPUs.

• Multi-Instance GPU (MIG) — Hardware-level partitioning on A100/H100. Creates isolated GPU instances with dedicated memory and compute. Best for inference and small training jobs. Limitation: fixed partition sizes (1g.5gb, 2g.10gb, 3g.20gb, 4g.20gb, 7g.40gb on A100).
• Time-slicing — Software-level sharing via NVIDIA's time-slicing feature. Multiple pods share one GPU with time-multiplexed access. No memory isolation — one pod can OOM-kill others. Suitable for development and interactive notebooks, not production training.
• Whole GPU — One pod gets one entire GPU. Maximum performance and isolation. Required for large training jobs. Most expensive per-workload.

Our recommendation: MIG for inference workloads, whole GPUs for training, and time-slicing only for development namespaces with clear "no SLA" expectations.

Fair Queuing with Kueue

When demand exceeds GPU capacity (it always does), you need fair queuing. Kueue provides:

• ClusterQueues — Define total GPU capacity and fair-sharing policies
• LocalQueues — Per-namespace queues that draw from cluster capacity
• Cohorts — Allow borrowing unused capacity across teams
• Preemption — Lower-priority workloads yield to higher-priority ones

Define priority classes: interactive (highest — notebooks), production-inference (high), training (medium), experimentation (low — preemptable). Teams get guaranteed minimum GPU hours, with ability to burst into unused capacity.

Topology-Aware Scheduling

For multi-GPU training jobs, GPU topology is critical. Two A100s connected via NVLink deliver ~600 GB/s bandwidth. Two A100s across PCIe deliver ~64 GB/s. A distributed training job that spans NVLink boundaries will be 10x slower for communication-heavy workloads.

Use Node Feature Discovery to label GPU topology. Write scheduling constraints that keep multi-GPU jobs on the same NVLink domain. For large jobs, use the topology-aware scheduling plugin in the scheduler.

Monitoring and Chargeback

GPU utilization monitoring is essential for multi-tenancy. Deploy DCGM Exporter for per-GPU metrics: utilization, memory usage, temperature, power consumption. Build Grafana dashboards per namespace. Implement chargeback based on GPU-hours consumed — this changes team behavior faster than any quota policy.

What Open Empower Delivers

We design and implement multi-tenant GPU infrastructure on OpenShift AI and vanilla Kubernetes. From MIG configuration to Kueue scheduling policies to topology-aware placement — GPU infrastructure that serves multiple teams efficiently and fairly.