A Secure AI Reference Architecture for Regulated Enterprises: What a Real-World Production Stack Looks Like

Every vendor publishes AI reference architectures. They're beautiful diagrams with their products at the centre and hand-waving at the edges. They show what to buy, not what to build. They omit the hard parts: security boundaries, data classification enforcement, compliance evidence generation, and what happens when things go wrong.

This article presents a real-world AI reference architecture for a regulated European enterprise. It's based on actual production deployments — anonymised and generalised, but every component has been battle-tested. It covers the full stack: from GPU hardware to application layer, with security controls and compliance mapping at every level.

Architecture Overview

The stack is organised in five layers. Each layer has specific technology choices, security controls, and compliance responsibilities:

Layer 5: Application Layer

Layer 1: Foundation — Where It Runs

The foundation choice is the most consequential decision for regulated enterprises:

Option A: Sovereign/Neocloud Provider

For enterprises where data residency is a hard requirement (financial services under DORA, healthcare under GDPR, government):

• Providers: OVHcloud, Scaleway, IONOS, nscale, Hetzner, OpenNebula-based private clouds
• Advantages: EU data residency guaranteed, no CLOUD Act exposure, GDPR-compliant by design, often cheaper GPU pricing
• Trade-offs: Smaller ecosystem, fewer managed services, less automation maturity than hyperscalers

Option B: Hyperscaler with Controls

For enterprises that can use hyperscalers with appropriate controls:

• Providers: AWS (EU regions), Azure (EU regions), GCP (EU regions)
• Required controls: Data residency configuration, encryption with customer-managed keys, VPC isolation, PrivateLink for all services, audit logging to customer-owned storage
• Trade-offs: CLOUD Act exposure (mitigated by encryption), vendor lock-in risk, higher GPU pricing

Option C: On-Premise

For maximum control (defence, intelligence, critical infrastructure):

• Hardware: Dell PowerEdge + NVIDIA H100/H200, HPE ProLiant, Lenovo ThinkSystem
• Advantages: Complete sovereignty, no third-party access, custom security controls
• Trade-offs: CapEx model, hardware lifecycle management, physical security responsibility

Layer 2: Infrastructure — Compute, Storage, Network

GPU Compute

Node pool architecture

Storage Architecture

• Model weights: Block storage (Ceph RBD, EBS, Azure Disk) — fast read for model loading, replicated for HA
• Training datasets: Object storage (MinIO, S3, Azure Blob) — scalable, versioned, with access logging
• Inference cache: NVMe local storage on GPU nodes — fastest possible access for KV-cache and model shards
• Audit logs: Append-only object storage in a separate account/project — immutable, retention-policy enforced

Network Architecture

• CNI: Cilium (preferred — eBPF-based, supports NetworkPolicy + observability) or Calico
• Ingress: Traefik or NGINX with TLS termination, rate limiting, and WAF rules
• Egress: Egress gateway with URL-level filtering. AI workloads cannot reach arbitrary internet endpoints
• Service mesh: Istio or Cilium mesh for mTLS between all services. Zero-trust network within the cluster
• Training network: Dedicated RDMA/InfiniBand network for multi-node training. Isolated from inference and platform traffic

Layer 3: Kubernetes Platform — Multi-Tenancy and Security

Namespace Architecture

# Namespace strategy for regulated AI workloads
Namespaces:
  ai-inference-prod:     # Production inference serving
    PSS: restricted
    NetworkPolicy: default-deny, allow only from ingress
    ResourceQuota: GPU limits per tenant
    RBAC: inference-service-account (read-only model store)

  ai-inference-staging:  # Staging/canary inference
    PSS: restricted
    NetworkPolicy: default-deny
    ResourceQuota: lower GPU limits
    RBAC: staging-service-account

  ai-training:           # Model training jobs
    PSS: baseline (needs some flexibility for GPU drivers)
    NetworkPolicy: allow dataset store, deny internet
    ResourceQuota: GPU limits, job time limits
    RBAC: training-service-account (read datasets, write models)

  ai-platform:           # Platform services (gateway, registry)
    PSS: restricted
    NetworkPolicy: allow from inference namespaces
    RBAC: platform-admin (limited scope)

  monitoring:            # Observability stack
    PSS: restricted
    NetworkPolicy: allow scraping from all namespaces
    RBAC: monitoring-readonly

  security:              # Falco, Kyverno, cert-manager
    PSS: privileged (Falco needs kernel access)
    NetworkPolicy: allow alerting egress only
    RBAC: security-admin

Security Controls

• Pod Security Standards: restricted on all namespaces except security (Falco) and training (GPU driver requirements)
• RBAC: Per-namespace service accounts with minimum permissions. No cluster-admin for human users — break-glass procedure for emergencies
• Admission control: Kyverno policies — signed images only, SBOM required, no privileged containers, resource limits required
• Runtime security: Falco with custom rules for AI workloads (detect unusual model file access, unexpected network connections from inference pods, GPU resource anomalies)
• Secret management: External Secrets Operator + HashiCorp Vault. No Kubernetes native secrets for sensitive data. Dynamic database credentials with automatic rotation

Layer 4: AI/ML Platform

• Model serving: vLLM for LLM inference (batching, PagedAttention, quantisation support). Triton for multi-framework serving (ONNX, TensorRT, PyTorch)
• Model registry: MLflow or Kubeflow Model Registry — versioned models with metadata, lineage, and deployment history
• Training: Kubeflow Training Operator for distributed training jobs. Supports PyTorch, TensorFlow, and JAX
• Feature store: Feast for feature management — consistent features between training and inference
• Experiment tracking: MLflow for experiment comparison, hyperparameter logging, and model performance metrics

Layer 5: Application — AI Gateway and Governance

The application layer is where business logic meets AI capability:

• AI Gateway: Centralised routing for all AI requests. Data classification enforcement (regulated data → self-hosted models only). Cost attribution per team/project. Rate limiting. Prompt/response logging for audit
• Action tier enforcement: Tier 1–4 system (see our agentic AI guardrails article) enforced at the gateway level
• Output validation: Schema validation, hallucination detection, PII scanning on outputs before they reach downstream systems
• Human review UI: For Tier 3/4 actions — AI proposes, human approves, system executes

Compliance Mapping

• DORA: ICT risk management (Layer 3 security controls), incident reporting (Layer 2 monitoring + Layer 5 audit), resilience testing (tabletop exercises + chaos engineering), third-party risk (Layer 1 provider assessment)
• NIS2: Security measures (Layers 2-3), incident handling (Layer 2 monitoring), supply chain (SBOM at Layer 4), business continuity (GitOps recovery at Layer 3)
• CRA: SBOM (Layer 4 build pipeline), vulnerability handling (Layer 3 scanning), security by design (all layers)
• EU AI Act: Risk management (Layer 5 action tiers), transparency (Layer 5 audit logging), human oversight (Layer 5 human-in-the-loop), accuracy (Layer 5 output validation)
• GDPR: Data residency (Layer 1 provider choice), data classification (Layer 5 gateway), access control (Layer 3 RBAC), audit (all layers)

What Open Empower Delivers

We design, implement, and operate this reference architecture for regulated enterprises:

• Architecture design: Layer-by-layer design decisions based on your regulatory requirements, data sensitivity, and team capabilities
• Foundation selection: Sovereign cloud, hyperscaler, or on-premise — evaluated against your specific regulatory and performance requirements
• Kubernetes platform: Multi-tenant namespace architecture, security controls, GitOps, and observability
• AI/ML platform: Model serving, training infrastructure, model registry, and feature store
• Governance layer: AI gateway, action tier enforcement, output validation, and compliance evidence generation

This isn't a diagram — it's a deployable architecture. Let's build yours.