Kubernetes DR: Active-Passive vs Active-Active Failover Patterns

Kubernetes doesn't solve disaster recovery — it just changes where the complexity lives. A single-cluster Kubernetes deployment is a single point of failure, regardless of how many replicas you run. When the cluster itself fails — cloud provider outage, region-level disaster, catastrophic misconfiguration — everything running on it goes down. For business-critical AI platforms, you need multi-cluster DR.

DR Architecture Patterns

Active-Passive

Primary cluster handles all traffic. Standby cluster has infrastructure deployed but workloads scaled to zero or minimal replicas. On failover: scale up standby, redirect traffic. RTO: 15-60 minutes depending on stateful workload complexity. Cost: ~30-50% additional infrastructure for the passive cluster.

Active-Active

Both clusters handle traffic simultaneously. Global load balancer distributes requests. On failure of one cluster, the other absorbs all traffic. RTO: near-zero for stateless workloads (just DNS/LB update). Cost: ~100% additional infrastructure. Complexity: data consistency across clusters.

Active-Warm

Hybrid approach. Both clusters are active, but the secondary runs a reduced workload (perhaps read-only queries or non-critical services). On failover, promote to full capacity. RTO: 5-15 minutes. Cost: 50-70% additional infrastructure. Best balance for most AI platforms.

Stateful Workload Recovery

Stateless services fail over easily. Stateful workloads are the hard part:

• Databases — PostgreSQL with streaming replication across clusters. pg_basebackup for initial sync, WAL shipping for continuous replication. Patroni for automated failover.
• Object storage — MinIO with bucket replication or cloud-native cross-region replication (S3 CRR). Model artifacts, training data, and checkpoints.
• Message queues — Kafka MirrorMaker 2 for cross-cluster topic replication. Or use a cloud-native queue (SQS, Pub/Sub) that's inherently multi-region.
• Model registries — MLflow with shared backend storage. Replicate the artifact store, not the metadata database.

Velero for Kubernetes State

Velero backs up Kubernetes resources and persistent volumes. Schedule regular backups: hourly for namespaces with active development, daily for stable workloads. Store backups in a different region/cloud than your primary cluster. Test restoration monthly.

Velero limitations: it backs up Kubernetes state, not application state. Your database needs its own backup strategy. Velero handles: deployments, configmaps, services, CRDs, RBAC — the cluster configuration that would take hours to recreate manually.

Automated Failover

DNS-based failover is the simplest approach:

• Health checks on primary cluster endpoints (every 10 seconds)
• On failure detection (3 consecutive failures): update DNS to point to DR cluster
• DNS TTL of 60 seconds means traffic shifts within ~2 minutes
• ArgoCD in DR cluster is already synced — applications deploy from the same Git repositories

For faster failover, use a global load balancer (AWS Global Accelerator, Cloudflare Load Balancing) with health-check-based routing. Failover is instantaneous — no DNS propagation delay.

DR Testing

A DR plan that hasn't been tested is just a wish. Schedule quarterly DR tests:

• Simulate primary cluster failure (cordon all nodes, disable ingress)
• Execute failover procedure
• Verify all services functional in DR cluster
• Measure actual RTO and RPO against targets
• Document failures and improve the procedure

What Open Empower Delivers

We design and implement multi-cluster DR for Kubernetes AI platforms. From architecture selection (active-passive, active-active, active-warm) to stateful workload replication to automated failover — resilience that's tested and proven.