Luca BertonPlatform engineering has a clear mission: reduce cognitive load on developers by providing self-service infrastructure with sensible defaults. For web application teams, this is well-understood — golden paths for deploying containers, databases, and APIs. For AI teams, it's a different problem entirely.
AI workloads don't look like web workloads. A data scientist training a model needs GPU allocation, experiment tracking, dataset versioning, and a way to compare runs. An ML engineer deploying inference needs model registries, A/B serving, canary rollouts, and latency monitoring. An agent developer needs tool orchestration, guardrails, memory persistence, and human-in-the-loop controls. None of these exist in a standard internal developer platform.
The result: AI teams build their own infrastructure. Every team reinvents experiment tracking. Every project has a different model serving approach. GPU allocation is a spreadsheet. Compliance is an afterthought. The platform team built a beautiful golden path — and the AI teams can't use it.
AI platform engineering fixes this. It extends the internal developer platform with purpose-built capabilities for ML, LLM, and agent workloads — so AI teams get the same self-service experience that application teams already enjoy.
What Makes AI Workloads Different
Before designing the platform, understand why AI workloads break standard platform assumptions:
- GPU is the bottleneck, not CPU: A web app needs 0.5 CPU and 512MB RAM. A model training job needs 4x A100 GPUs and 320GB GPU memory for 6 hours. Standard Kubernetes scheduling doesn't understand GPU topology, NVLink affinity, or time-bounded allocation
- Workloads are stateful and long-running: A training job runs for hours or days. If it's preempted, hours of compute are wasted. Checkpointing, spot instance tolerance, and priority scheduling are essential — not optional
- The workflow is iterative, not linear: Web development: code → test → deploy. ML development: explore data → train → evaluate → retrain → tune → evaluate → deploy → monitor for drift → retrain. The platform must support experimentation loops, not just deployment pipelines
- Data is a first-class concern: Web apps read from databases. ML workloads need dataset versioning, data lineage, feature stores, and data quality checks. The training data is as important as the code
- Models are not containers: A model is a binary artifact with metadata (metrics, parameters, dataset version, training code hash). It needs a registry, versioning, promotion stages, and rollback — but the semantics are different from container image registries
- Compliance is per-model, not per-service: Under the EU AI Act, each AI system has a risk classification. The platform must track which models are high-risk, what data they were trained on, and provide audit evidence per model — not per deployment
The AI Platform Stack
An AI internal developer platform has five layers, each building on the one below:
Layer 5: Self-Service Portal
Web UI / CLI / API — "Give me a GPU workspace for 8 hours" · "Deploy model v2.3 to staging" · "Show me my team's GPU spend this month"
Layer 4: AI Workflow Services
Experiment Tracking (MLflow / W&B) · Model Registry + Promotion · Feature Store (Feast) · Agent Orchestration + Guardrails · Dataset Versioning (DVC / LakeFS)
Layer 3: AI Compute Services
GPU Scheduling (Kueue / Volcano / Yunikorn) · Model Serving (vLLM / Triton / KServe) · Notebook Environments (JupyterHub) · Training Orchestration (Kubeflow / Ray) · Vector Databases (pgvector / Milvus)
Layer 2: Platform Foundation
Kubernetes (OpenShift / EKS / GKE) · GitOps (ArgoCD) · Observability (Prometheus / Grafana) · Policy (Kyverno / OPA) · Secrets (Vault) · Service Mesh (Istio)
Layer 1: Infrastructure
GPU Nodes (A100 / H100 / L4) · Storage (NFS / S3 / Ceph) · Networking (RDMA / InfiniBand for training) · Sovereignty Zones (EU data residency)
Kubernetes Recipes
A practical guide for container orchestration and deployment by Grzegorz Stencel & Luca Berton (Apress).
Watch on Skillshare →Golden Paths for AI Teams
Golden paths for AI teams look different from application golden paths. Here are the five essential ones:
Golden Path 1: Experiment-to-Model
For data scientists running experiments:
Step 1: Request GPU workspace
--gpu 2xA100 \
--duration 8h \
--team ml-team \
--project fraud-detection-v3
- ✓ JupyterHub environment provisioned
- ✓ 2x A100 GPUs allocated via Kueue
- ✓ MLflow experiment auto-created
- ✓ Team's shared dataset volume mounted
- ✓ Cost tracking started (2x A100 @ 3.50 EUR/h)
Step 2: Train and track (inside notebook)
mlflow.autolog()
model = train_model(data, hyperparams)
# Metrics, params, artifacts auto-logged
# Dataset hash recorded for lineage
Step 3: Register model
--name fraud-detector \
--version 3.1 \
--experiment-run abc123 \
--risk-class high # EU AI Act classification
- ✓ Model stored in registry with full lineage
- ✓ Compliance metadata attached
- ✓ Automated bias + fairness checks triggered
- ✓ Security scan on model artifacts
Step 4: Workspace auto-reclaimed after 8h
- ✓ GPU released back to pool
- ✓ Notification: "Workspace expired. Results in MLflow."
Golden Path 2: Model-to-Production
For ML engineers deploying models:
--name fraud-detector \
--version 3.1 \
--target staging \
--serving vllm \
--replicas 2 \
--gpu L4
- ✓ KServe InferenceService created
- ✓ Model loaded onto L4 GPU serving nodes
- ✓ Canary configuration: 10% traffic for 2 hours
- ✓ Automated tests run against staging endpoint
- ✓ Latency/accuracy monitoring dashboards auto-created
- ✓ Rollback trigger: if p99 latency > 200ms or accuracy < 0.95
After staging validation:
--name fraud-detector \
--version 3.1 \
--from staging \
--to production
- ✓ Progressive rollout: 10% → 25% → 50% → 100%
- ✓ Old model kept warm for instant rollback
- ✓ Compliance evidence generated (deployment record)
Golden Path 3: Agent Development
For teams building AI agents with tools and guardrails:
--name customer-support-agent \
--model gpt-4o \
--tools "crm-lookup,order-status,escalate-human" \
--guardrails "pii-filter,cost-limit-50,no-refund-authority" \
--memory redis \
--team support-engineering
Agent runtime provisioned with:
- ✓ LLM gateway with rate limiting and cost caps
- ✓ Tool registry with RBAC (agent can call CRM, not modify it)
- ✓ Guardrail chain (PII filter → cost check → authority check)
- ✓ Redis-backed conversation memory
- ✓ Human-in-the-loop escalation webhook
- ✓ Full request/response audit logging
Agent configuration managed via Git (GitOps):
customer-support/
agent.yaml # Model, tools, guardrails config
tools/
crm-lookup.yaml # Tool definition + RBAC
order-status.yaml
guardrails/
pii-filter.yaml # Regex + NER-based PII detection
cost-limit.yaml # Max spend per conversation
tests/
scenarios.yaml # Automated test conversations
Agent guardrails and trust boundaries are critical — the platform enforces them, not the agent developer.
Golden Path 4: LLM Fine-Tuning
--base-model llama-3.1-8b \
--dataset s3://datasets/support-conversations-v2 \
--method lora \
--gpu 4xA100 \
--epochs 3 \
--team nlp-team
- ✓ Fine-tuning job submitted to Kueue queue
- ✓ Priority: below production inference, above experiments
- ✓ Estimated wait: 2 hours (GPU pool utilisation: 78%)
- ✓ Estimated training time: 4 hours
- ✓ Estimated cost: 56 EUR
- ✓ Auto-evaluation against benchmark suite on completion
- ✓ Results in MLflow with full reproducibility metadata
Golden Path 5: Data Pipeline
--name customer-features \
--source postgres://transactions \
--features "avg_spend_30d,tx_count_7d,last_login_days" \
--schedule "0 6 * * *" \
--store feast \
--team ml-team
- ✓ Airflow/Dagster DAG generated
- ✓ Feature computation pipeline deployed
- ✓ Feast feature store updated daily at 06:00
- ✓ Data quality checks (Great Expectations) auto-configured
- ✓ Lineage tracked: source → transformation → feature store → model
GPU as a Managed Service
GPU management is the hardest part of AI platform engineering. Make it a managed service that teams consume, not infrastructure they manage:
- GPU pools by tier: Training pool (A100/H100 — high priority, preemption-protected), inference pool (L4/T4 — always-on, auto-scaled), experiment pool (shared A100 — best-effort, preemptible). Teams request from pools, not specific hardware
- Time-bounded allocation: No indefinite GPU reservations. Every allocation has a TTL. Expired allocations auto-reclaim. Time-based scheduling for predictable workloads
- Queue management: Kueue or Volcano for fair scheduling across teams. Per-team quotas (ML team: 16 A100-hours/day). Priority levels (production > fine-tuning > training > experiments). Queue visibility: "Your job is #3, estimated start: 45 minutes"
- Cost transparency: Per-team, per-project GPU cost dashboards updated in real-time. Budget alerts at 80% and 100% of monthly allocation. Chargeback or showback — teams see what their AI costs
- MIG (Multi-Instance GPU): Split A100s into smaller instances for lightweight workloads. A data scientist running exploratory analysis doesn't need a full A100 — a 1/7 MIG slice (10GB) is enough and 7x more efficient
Automating Azure DevTest Labs
Automate lab management and integrate with CI/CD pipelines.
Start on Pluralsight →Observability for AI Workloads
Standard APM doesn't cover AI workloads. The platform must provide AI-specific observability:
- GPU metrics: Utilisation, memory usage, temperature, power draw, MIG allocation — per pod, per team, per project. DCGM Exporter → Prometheus → Grafana dashboards
- Model performance: Inference latency (p50/p95/p99), throughput (tokens/sec for LLMs), accuracy/F1 on live traffic, drift detection (feature drift + prediction drift)
- Cost tracking: GPU-hours consumed per team, cost per inference request, training job cost, total AI spend with trend analysis
- Experiment tracking: MLflow/W&B dashboards showing experiment progress, hyperparameter search, model comparison across runs
- Agent observability: Tool call traces, guardrail trigger rates, escalation frequency, conversation quality metrics, cost per conversation
Compliance Built Into the Platform
For regulated enterprises under DORA, NIS2, and the EU AI Act, compliance can't be a separate process. It must be embedded in the platform:
- Model cards auto-generated: Every registered model gets a model card with: training data description, performance metrics, bias evaluation results, intended use, limitations, risk classification
- Audit trail: Full lineage from data → training → evaluation → deployment → monitoring. Every step logged, timestamped, attributed to a user. Immutable audit log
- Policy gates: High-risk models (EU AI Act Article 6) cannot be deployed without: bias evaluation passing thresholds, human review approval, compliance officer sign-off. Enforced by Kyverno policies on the deployment pipeline
- Data residency: Sovereignty zones enforced at the platform level. Training data tagged with residency requirements. GPU workloads scheduled only on compliant nodes
AI Readiness Assessment
Fixed-fee, 3-4 week infrastructure and compliance audit with a concrete roadmap. No big-consultancy overhead.
See Scope & Pricing →Build vs Buy: The Platform Composition
You don't build an AI platform from scratch. You compose it from best-in-class components:
AI Platform Component Map
| Component | OSS Option | Managed Option |
|---|---|---|
| GPU Scheduling | Kueue, Volcano | OpenShift AI |
| Experiment Tracking | MLflow | W&B, Neptune |
| Model Registry | MLflow | Vertex AI, SageMaker |
| Model Serving | vLLM, KServe | OpenShift AI, Vertex |
| Notebook Environments | JupyterHub | SageMaker Studio |
| Training Orchestration | Kubeflow, Ray | Vertex Pipelines |
| Feature Store | Feast | Tecton, Vertex FS |
| Vector Database | pgvector, Milvus | Pinecone, Weaviate |
| Agent Framework | LangChain, CrewAI | Azure AI Agent |
| Data Versioning | DVC, LakeFS | Pachyderm |
| Pipeline Orchestration | Airflow, Dagster | Managed Airflow |
What Open Empower Delivers
- AI platform assessment: Evaluate your current AI infrastructure maturity. Identify gaps in GPU management, model lifecycle, experiment tracking, and compliance. Produce a prioritised roadmap
- Platform architecture: Design the five-layer AI platform stack for your organisation. Component selection (build vs buy), integration architecture, and golden path definitions
- GPU-as-a-service: Implement Kueue/Volcano scheduling with per-team quotas, time-bounded allocation, MIG configuration, and cost transparency dashboards
- ML/LLM workflow: MLflow experiment tracking, model registry with promotion stages, KServe/vLLM serving with canary deployments, and automated evaluation pipelines
- Agent platform: Agent development golden path with tool registries, guardrail enforcement, memory management, and human-in-the-loop controls
- Compliance automation: EU AI Act model cards, DORA audit trails, automated bias evaluation, and policy-gated deployment for high-risk AI systems
Your AI teams deserve the same self-service experience your application teams already have. Let's build the platform that gives it to them.