The Infrastructure Layer of Cloud: What Enterprises Get Wrong and How to Fix It Before AI Makes It Worse

The infrastructure layer of cloud was designed for web applications. Stateless HTTP services that scale horizontally, consume modest CPU and memory, and are tolerant of network latency. The entire cloud ecosystem — instance types, storage tiers, networking models, pricing — optimises for this workload pattern.

AI workloads are different in almost every dimension. They're compute-intensive (GPU, not CPU). They're memory-hungry (model weights measured in gigabytes, not megabytes). They're storage-dependent (training datasets measured in terabytes). They're network-sensitive (distributed training requires low-latency, high-bandwidth interconnects). And they're stateful (models, checkpoints, and caches must persist across restarts).

Enterprises that built their cloud infrastructure for web applications and then try to run AI workloads on it discover — painfully — that their infrastructure layer needs fundamental changes. This article maps those changes.

Compute: The GPU Revolution

What's Wrong Today

Most enterprise Kubernetes clusters have zero GPU nodes. GPU instances are provisioned ad hoc by data science teams using separate cloud accounts, Jupyter notebooks, and SageMaker/Vertex AI — completely outside the platform engineering team's governance.

This creates:

• Shadow AI infrastructure: GPU spend invisible to central IT. No security controls. No cost governance. No SBOM. No compliance evidence
• GPU waste: Data scientists provision large GPU instances, use them for a few hours, forget to terminate them. Average GPU utilisation in enterprise: 15–30%
• No multi-tenancy: Each team runs their own GPU nodes because there's no shared GPU platform. The same H100 that costs $30K/year sits idle 70% of the time

What It Should Look Like

GPU compute architecture for enterprise AI

Storage: Tiered for AI Workloads

What's Wrong Today

Enterprise storage is typically a single tier: general-purpose block storage (AWS gp3, Azure Standard SSD) adequate for database workloads and application state. AI workloads need multiple storage tiers with very different performance characteristics:

What It Should Look Like

• Hot tier (NVMe local): GPU node-local NVMe storage for model weights actively being served, KV-cache, and intermediate training checkpoints. Fastest possible I/O. Ephemeral — data reconstructed from model registry on pod startup. 1–4 TB per GPU node
• Warm tier (block storage): Network-attached block storage (Ceph RBD, EBS io2, Azure Ultra Disk) for model registry, feature store, and active datasets. Persistent, replicated, snapshotted. 10–100 TB
• Cold tier (object storage): S3-compatible object storage (MinIO, S3, GCS) for dataset archives, training logs, experiment artifacts, and audit records. Cheap, scalable, versioned. 100 TB+
• Archive tier: Glacier-class storage for regulatory retention. Training data lineage, model provenance records, compliance evidence. WORM (Write Once Read Many) for immutability. Retained per regulatory requirement (DORA: 5 years, GDPR: varies)

Networking: Built for AI Traffic Patterns

What's Wrong Today

Enterprise cloud networking is built for north-south traffic (client → server) with modest east-west traffic (service → service). AI workloads generate massive east-west traffic that the existing network architecture can't handle:

• Distributed training: Multi-node training synchronises gradients across GPUs. AllReduce operations can generate 100+ Gbps of east-west traffic per training job. Standard 10Gbps pod networking is a bottleneck
• Model loading: Loading a 70B parameter model from storage to GPU memory transfers ~140 GB. On 10Gbps networking, that's ~2 minutes. On InfiniBand (200 Gbps), it's ~6 seconds
• RAG pipelines: Retrieval-Augmented Generation queries a vector database for every inference request. Latency between inference pod and vector DB directly impacts response time

What It Should Look Like

• Training network: Dedicated RDMA/InfiniBand or RoCE (RDMA over Converged Ethernet) network for GPU-to-GPU communication during distributed training. Isolated from inference and platform traffic. 200–400 Gbps per node
• Inference network: Low-latency (<1ms) connectivity between inference pods, vector databases, and caching layers. Cilium with eBPF for efficient L3/L4 processing. Service mesh for mTLS without latency overhead
• Storage network: Dedicated network for model loading and dataset access. Separate from application traffic to prevent I/O operations from affecting inference latency
• Egress control: All outbound traffic through egress gateways with URL-level filtering. AI workloads cannot reach arbitrary internet endpoints — critical for preventing data exfiltration and satisfying data residency requirements

Capacity Planning: AI Workloads Are Different

Traditional capacity planning based on CPU and memory doesn't work for AI:

• GPU capacity: Plan in GPU-hours, not instance count. Track utilisation at the MIG instance level, not the node level. A node reporting 50% GPU utilisation might have 4 fully utilised MIG instances and 3 completely idle ones
• VRAM is the bottleneck: Model size determines minimum GPU memory. A 70B model needs ~140 GB VRAM (FP16). Two H100s (160 GB total) can serve it with tensor parallelism. This is a hard constraint — you can't swap VRAM to disk without catastrophic performance impact
• Burst capacity: Training jobs are bursty — a fine-tuning job might need 8 GPUs for 6 hours, then nothing for a week. Build for burst: spot/preemptible GPU instances, cross-cloud bursting, or on-demand cloud GPU alongside on-premise base capacity
• Growth planning: AI compute demand typically grows 3–5x year-over-year in early adoption. Plan GPU procurement 6–12 months ahead — GPU supply chains have long lead times

The Hybrid Cloud Reality

Most regulated enterprises will run a hybrid architecture:

• On-premise: Base GPU capacity for steady-state inference (the workload that runs 24/7). Sensitive training workloads that can't leave the building. Model registry and feature store
• Cloud: Burst GPU capacity for training (spot/preemptible instances for cost optimisation). Non-sensitive workloads. Disaster recovery
• Sovereign cloud: For workloads that must stay in the EU but don't require on-premise. Data residency with managed infrastructure

The Kubernetes layer abstracts this: same namespaces, same RBAC, same NetworkPolicies, same GitOps — regardless of whether the underlying nodes are on-premise or cloud. Federation or multi-cluster management (Submariner, Skupper, or cloud-specific tools) connects the clusters.

What Open Empower Delivers

We design and implement AI-ready infrastructure layers for regulated enterprises:

• GPU platform design: Shared GPU pools with multi-tenancy, scheduling, and cost governance
• Storage architecture: Tiered storage for model serving, training, and regulatory retention
• Network design: Training networks, inference networks, and egress controls for AI workloads
• Capacity planning: GPU-hours modelling, VRAM planning, and procurement strategy
• Hybrid cloud architecture: On-premise + cloud + sovereign cloud with unified Kubernetes management

Your infrastructure layer was built for web apps. AI demands more. Let's rebuild the foundation.