Choosing Your Network Infrastructure Provider for AI: What Regulated Enterprises Need Beyond Bandwidth

When enterprises evaluate infrastructure providers for AI workloads, compute (GPUs) and storage get most of the attention. Network infrastructure is treated as an afterthought — "we'll use whatever the cloud provider gives us." This is a mistake that surfaces as performance bottlenecks, compliance violations, and security gaps once AI workloads hit production.

Network infrastructure for AI is fundamentally different from network infrastructure for web applications. Distributed training generates east-west traffic volumes that would overwhelm a traditional cloud VPC. Data sovereignty regulations restrict where traffic can flow. Inference latency requirements demand sub-millisecond connectivity. And network observability must produce audit evidence, not just performance metrics.

What AI Workloads Demand From the Network

Distributed Training: The Bandwidth Problem

Multi-node training synchronises model gradients between GPUs after every batch. For a 70B parameter model training on 8 nodes with 8 GPUs each (64 GPUs total), each synchronisation step transfers ~140 GB across the network. At 100 batches per second (typical for large models), that's 14 TB/second of aggregate cross-node traffic.

• Standard 25 Gbps cloud networking: Each gradient sync takes ~45 seconds. Training becomes network-bound — GPUs sit idle waiting for network transfers. Effective GPU utilisation drops to 20–40%
• 100 Gbps RDMA (RoCEv2): Gradient sync drops to ~11 seconds. Better, but still significant overhead
• 200 Gbps InfiniBand (NDR): Gradient sync in ~5.6 seconds. GPU utilisation reaches 70–85%. This is the target for serious training workloads
• 400 Gbps InfiniBand (NDR400): Available on latest hardware. Near-zero network overhead for most training configurations

The network choice directly determines your GPU ROI. Running $30K/year H100 GPUs on $5K/year 25 Gbps networking means paying for GPU capacity you can't use. The network is the bottleneck, and the GPU spend is wasted.

Inference: The Latency Problem

Inference latency directly impacts user experience and throughput:

• Model loading: When a new model version deploys, 140 GB of weights transfer from storage to GPU memory. Network latency determines cold-start time
• RAG retrieval: Every inference request that uses Retrieval-Augmented Generation queries a vector database. Network round-trip between inference pod and vector DB adds to response latency. At 100 requests/second, even 5ms of additional network latency means 500ms of cumulative delay per second
• Multi-model pipelines: Complex AI workflows chain multiple models (e.g., embedding → retrieval → generation → validation). Each hop adds network latency. A 4-hop pipeline with 2ms per hop adds 8ms to every request

Provider Evaluation Criteria

1. GPU Interconnect Technology

• Must have: RDMA support (RoCEv2 minimum) for GPU-to-GPU communication within a training job
• Preferred: InfiniBand (NDR 200 Gbps+) for serious multi-node training
• Evaluate: Is the high-speed interconnect available between all GPU nodes, or only within a "placement group" or "GPU cluster"? Many providers offer high-speed interconnects only within a rack or availability zone
• Ask: What's the oversubscription ratio? A 200 Gbps port with 4:1 oversubscription delivers 50 Gbps under load

2. Network Isolation and Segmentation

• Must have: VPC/VNet isolation between tenants. No shared network segments with other customers
• Must have: Support for Kubernetes NetworkPolicies (the CNI must enforce them — not all provider CNIs do)
• Preferred: Separate physical or logical networks for training traffic, inference traffic, and management traffic
• Evaluate: Can you create network isolation between namespaces/projects within your Kubernetes cluster at the infrastructure level (not just at the CNI level)?

3. Egress Control and Data Sovereignty

For regulated enterprises, controlling where traffic goes is as important as controlling where it comes from:

• Must have: Ability to restrict all egress to approved destinations. AI workloads that can reach arbitrary internet endpoints are a data exfiltration risk
• Must have: Traffic inspection capability (or the ability to deploy your own inspection) without routing regulated data through the provider's inspection infrastructure
• Preferred: Provider-level egress filtering that integrates with your Kubernetes egress gateway policies
• Critical for DORA/NIS2: Can you prove, with audit evidence, that regulated data never left the specified geographic region? Network flow logs must show this

4. DDoS Protection

• Must have: Volumetric DDoS protection at the network edge
• Evaluate carefully: Does the DDoS protection service inspect/decrypt your traffic? For regulated workloads with encrypted sensitive data, DDoS inspection that decrypts traffic may violate confidentiality requirements
• Preferred: DDoS protection that operates at L3/L4 (volumetric/protocol) without L7 inspection, plus separate application-layer protection (WAF) that you control

5. Private Connectivity

For hybrid deployments (on-premise + cloud):

• Must have: Private connectivity options (Direct Connect, ExpressRoute, Cloud Interconnect, or equivalent)
• Evaluate: Bandwidth options (1 Gbps is insufficient for AI; 10+ Gbps preferred), latency guarantees, and redundancy
• For multi-cloud: Cloud-to-cloud private connectivity (avoiding internet transit) for workload migration and disaster recovery

6. Network Observability

• Must have: VPC flow logs exportable to your own storage and SIEM
• Must have: DNS query logging (critical for detecting C2 communication and data exfiltration via DNS)
• Preferred: Integration with your Kubernetes-level observability (Cilium Hubble, Istio telemetry) for end-to-end request tracing
• For compliance: Flow logs must include source, destination, port, protocol, bytes transferred, and timestamp — the minimum for DORA ICT risk monitoring

Provider Comparison: Networking for AI

Hyperscalers

• AWS: EFA (Elastic Fabric Adapter) for RDMA within placement groups. Up to 400 Gbps. Strong VPC isolation. Transit Gateway for multi-VPC. Flow logs to S3/CloudWatch
• Azure: InfiniBand NDR on ND-series VMs. Strong for multi-node training. VNet with NSGs. ExpressRoute for hybrid. Flow logs to Log Analytics
• GCP: GPUDirect-TCPX for GPU communication. VPC with firewall rules. Cloud Interconnect for hybrid. Flow logs to BigQuery

Sovereign/Neocloud

• OVHcloud: vRack for private networking. InfiniBand available on GPU instances. EU data residency guaranteed. Growing Kubernetes support (MKS)
• Scaleway: Private Networks with VPC. GPU instances with local NVMe. EU-only infrastructure. Strong Kubernetes offering (Kapsule)
• nscale: EU-sovereign GPU cloud. High-bandwidth interconnects for training. Kubernetes-native. GDPR-compliant by design

On-Premise

• NVIDIA DGX/HGX: NVLink + NVSwitch for intra-node, InfiniBand for inter-node. Maximum performance, maximum control, maximum cost
• Dell PowerEdge + Networking: PowerSwitch for 100/400G Ethernet. SmartFabric for automated network provisioning. Integrates with Dell APEX for hybrid

Kubernetes Networking Layer

Regardless of provider, the Kubernetes networking layer must be configured for AI workloads:

• CNI choice: Cilium (preferred for eBPF performance, NetworkPolicy enforcement, and Hubble observability) or Calico (mature, well-tested)
• Multi-network support: Multus CNI for pods that need both standard Kubernetes networking and high-speed GPU interconnects (RDMA/InfiniBand as a secondary network)
• Service mesh: Istio or Cilium mesh for mTLS between services. Evaluate latency overhead — some service mesh configurations add 1–3ms per hop, which may be unacceptable for inference chains
• DNS: CoreDNS with caching. AI workloads that resolve many external endpoints (embedding APIs, vector databases, feature stores) benefit from aggressive DNS caching

For comprehensive Kubernetes networking patterns — Services, Ingress, NetworkPolicies, and multi-cluster connectivity — see Kubernetes Recipes.

What Open Empower Delivers

We evaluate and implement network infrastructure for AI workloads in regulated enterprises:

• Provider evaluation: Assess hyperscaler, sovereign, and on-premise networking against your specific AI workload patterns and regulatory requirements
• Network architecture: Training networks, inference networks, egress controls, and observability — designed for AI traffic patterns
• Kubernetes networking: CNI selection, NetworkPolicy design, service mesh configuration, and multi-network setup for GPU interconnects
• Compliance: Network flow logging, egress auditing, and data sovereignty evidence for DORA/NIS2/GDPR
• Hybrid connectivity: Private links between on-premise, cloud, and sovereign cloud with consistent security policies

Your network is either your AI platform's accelerator or its bottleneck. Let's make it an accelerator.