Multi-Dimensional Scale-Away: The Deployment Architecture Where Everything Shrinks to Zero When Not Needed

Enterprise AI infrastructure has a utilisation problem. GPU nodes purchased for $30K/year run at 15–30% average utilisation. Inference endpoints provisioned for peak load sit idle 80% of the time. RAG pipelines with vector databases keep warm replicas running 24/7 for workloads that peak during business hours. The monitoring stack for all of this consumes resources proportional to what it monitors, not to what's actually happening.

Traditional autoscaling addresses half of this: it adds capacity when demand rises. But it doesn't address the other half: removing capacity when demand disappears. "Scale to 2 replicas minimum" means paying for 2 replicas at 3 AM when nobody is using the system.

Multi-dimensional scale-away is an architecture where every layer of the AI stack can independently scale to zero — and wake up fast enough that users don't notice. Not just the inference pods, but the GPU nodes, the vector databases, the preprocessing pipelines, the caches, and the observability stack. Every component earns its resource consumption in real-time.

Why AI Workloads Need Scale-Away More Than Web Apps

Web application autoscaling is well-understood: add pods when CPU exceeds 70%, remove when it drops below 30%. The resources involved are cheap (CPU, memory) and the scaling speed is fast (new pods in seconds).

AI workloads are different in every dimension:

• Resources are expensive: A single H100 GPU costs $2.50–$3.50/hour. An idle H100 burns $21K–$30K/year. A cluster of 8 idle H100s is $170K–$240K/year in waste
• Startup is slow: Loading a 70B parameter model from storage to GPU memory takes 30 seconds to 3 minutes depending on network and storage speed. Cold-start latency is a real problem
• Demand is bursty: Enterprise AI usage follows business hours (9–17), batch processing schedules, and event-driven patterns. The 2 AM utilisation is near zero
• Components are coupled: An inference endpoint needs its vector database, its preprocessing pipeline, and its cache. Scaling one without the others doesn't work

The Five Dimensions of Scale-Away

Dimension 1: Inference Pod Scale-Away

The inference pods themselves — the containers running vLLM, Triton, or TGI — should scale to zero when there are no requests:

# KEDA ScaledObject for inference pod scale-away
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llm-inference-scaledobject
  namespace: ai-inference
spec:
  scaleTargetRef:
    name: llm-inference
  minReplicaCount: 0          # Scale to zero!
  maxReplicaCount: 8
  cooldownPeriod: 300          # Wait 5 min of idle before scaling down
  advanced:
    restoreToOriginalReplicaCount: false
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        metricName: inference_requests_pending
        query: sum(inference_queue_depth{namespace="ai-inference"})
        threshold: "1"         # Scale up when any request is pending
    - type: cron
      metadata:
        timezone: Europe/Amsterdam
        start: 0 8 * * 1-5     # Pre-warm at 8 AM weekdays
        end: 0 19 * * 1-5      # Allow scale-down after 7 PM
        desiredReplicas: "2"   # Keep 2 warm during business hours

The challenge: cold-start latency. A model that takes 90 seconds to load can't serve the first request of the day promptly. Solutions:

• Cron-based pre-warming: KEDA cron trigger spins up pods before business hours. First user arrives to warm pods
• Model caching on NVMe: GPU nodes with local NVMe store model weights. Loading from local NVMe (3–5 seconds) vs. network storage (30–90 seconds)
• Lightweight sentinel: A single small pod stays warm to handle trickle requests and triggers scale-up of full inference pods
• Request queuing: First request queues and triggers scale-up. Response arrives in ~30–90 seconds rather than timing out

Dimension 2: GPU Node Scale-Away

Empty GPU nodes (no inference pods scheduled) should be removed from the cluster entirely:

# Cluster Autoscaler configuration for GPU nodes
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-config
data:
  # Scale down GPU nodes after 10 minutes of zero utilisation
  scale-down-unneeded-time: "10m"
  # Don't scale down during business hours (inference pre-warmed)
  scale-down-non-empty-candidates-count: "0"
  # GPU node groups with specific settings
  nodes: |
    - name: gpu-h100-pool
      minSize: 0               # Can scale to zero nodes!
      maxSize: 8
      # Scale-down blocked during training jobs
      scaleDownBlockedByAnnotation: true

For cloud-hosted GPU nodes, this is direct cost savings: no node, no charge. For on-premise GPUs, scale-away is about power consumption — idle H100s still draw ~100W each. Shutting down nodes saves power and cooling costs.

Dimension 3: Supporting Services Scale-Away

Inference doesn't happen in isolation. The supporting services must also scale with demand:

• Vector database: Qdrant, Milvus, or Weaviate for RAG workloads. These can scale horizontally but rarely to zero (index loading time). Strategy: reduce to 1 compact replica overnight, scale during business hours
• Preprocessing pipeline: Document chunking, embedding generation, and data transformation. These are batch workloads that should run as Kubernetes Jobs (inherently scale-away — they complete and release resources)
• Cache layer: Redis/Valkey for prompt caching and session state. Scale replicas with inference pods. Persistent storage means cache survives scale-down; read replicas scale with load
• API gateway: The gateway itself should scale with request volume. Lightweight enough that 1 replica can handle trickle traffic; scales up for peak

Dimension 4: Observability Scale-Away

This is the dimension everyone forgets. Monitoring infrastructure typically runs at constant capacity regardless of what it monitors:

• Prometheus: Scrape interval doesn't change when workloads scale down. With 0 inference pods, Prometheus still scrapes every 15 seconds. Solution: Prometheus Agent mode with remote write — scale scraping capacity with workload count
• Log aggregation: Loki/Fluentd collectors run on every node. When GPU nodes scale to zero, their log collectors go with them (DaemonSet scales with nodes — this is automatic)
• Alerting: Alerts should be workload-aware. "Inference latency above 500ms" shouldn't fire when the inference deployment is at 0 replicas
• Dashboards: Grafana itself can be lightweight (1 small pod), but heavy queries on sleeping metrics should be avoided. Pre-computed recording rules reduce query cost

Dimension 5: Development/Staging Scale-Away

Non-production environments should be the most aggressive scale-away targets:

• Dev environments: Scale to zero after 30 minutes of inactivity. Developer triggers scale-up via CLI or Backstage IDP. Total cold-start: 2–3 minutes (acceptable for development)
• Staging: Scale to zero overnight and weekends. Pre-warm 30 minutes before the CI/CD window
• Load testing: Only exists during active load tests. Kubernetes Jobs that create, test, and destroy the environment

The Orchestration Layer

Multi-dimensional scale-away requires coordination. You can't scale down the vector database while inference pods still need it:

# Scale-away dependency graph
inference_pods:
  depends_on: [vector_db, cache, api_gateway]
  scale_to_zero_after: 5m idle
  cold_start: 30-90s (model loading)

vector_db:
  depends_on: []
  scale_to_minimum_after: 15m after last inference pod
  minimum: 1 replica (index must stay loaded)
  cold_start: 10-30s (index loading)

cache:
  depends_on: []
  scale_to_zero_after: 5m after last inference pod
  cold_start: 2-5s (connects to persistent storage)

api_gateway:
  depends_on: []
  scale_to_minimum_after: never (always 1 replica, lightweight)
  cold_start: <1s

gpu_nodes:
  depends_on: [inference_pods_at_zero]
  scale_to_zero_after: 10m after last pod evicted
  cold_start: 2-5m (cloud node provisioning)

# Wake-up chain (triggered by incoming request):
request -> api_gateway (always warm) ->
  queue request ->
  trigger inference pods (KEDA) ->
  trigger GPU node (cluster autoscaler, if needed) ->
  wait for model load ->
  serve request

Cost Impact

For a typical enterprise AI deployment (8 H100 GPUs, supporting infrastructure):

• Always-on (traditional): ~$280K/year GPU + ~$60K/year supporting infra = $340K/year
• Business-hours only (12h/day, weekdays): ~$100K/year GPU + ~$25K/year supporting = $125K/year (63% reduction)
• Demand-driven scale-away: ~$60K–$80K/year total (76–82% reduction). Actual spend correlates with actual usage

The savings fund the engineering investment to build the scale-away architecture within the first year.

What Breaks (and How to Fix It)

Health Checks and Readiness

Standard Kubernetes health checks don't account for model loading. A pod that's started but hasn't loaded its model will fail inference requests:

• Use startupProbe with generous timeout (120s+) for model loading
• Separate readiness from liveness: pod is alive (liveness) but not ready until model is loaded (readiness)
• KEDA shouldn't count pods as "scaled up" until they're ready, preventing premature scale-down decisions

Session State

Multi-turn AI conversations require session state. If the inference pod scales to zero between turns:

• Store conversation context in the cache layer (Redis/Valkey with persistence), not in pod memory
• KV-cache for the LLM can be serialised to Redis and reloaded when the pod wakes up (newer vLLM versions support this)

SLA Commitments

If your SLA promises <500ms response time, a 90-second cold start violates it. Solutions:

• Tiered SLA: guaranteed latency during business hours (pre-warmed), best-effort outside hours
• Sentinel pod: always-warm lightweight instance that handles low-volume traffic at acceptable latency
• Predictive scaling: ML-based prediction of when demand will return, pre-warming before users arrive

Implementation on Kubernetes

The Kubernetes ecosystem provides all the primitives:

• Pod scaling: KEDA (event-driven, supports Prometheus, cron, custom metrics, and many sources)
• Node scaling: Cluster Autoscaler or Karpenter (faster, more flexible node provisioning)
• Serverless inference: Knative Serving or KServe for request-driven scale-to-zero with queue-based activation
• Batch workloads: Kubernetes Jobs + Volcano/Kueue for training and preprocessing (inherent scale-away)
• Observability: Prometheus Operator with Agent mode, Grafana Loki, OpenTelemetry Collector

For the Kubernetes deployment patterns, Jobs, autoscaling, and resource management that underpin scale-away architectures, see Kubernetes Recipes.

What Open Empower Delivers

We design and implement multi-dimensional scale-away architectures for enterprise AI:

• Scale-away architecture: Dependency-aware scaling across inference, GPU nodes, supporting services, and observability
• Cold-start optimisation: Model caching, pre-warming strategies, sentinel pods, and predictive scaling
• Cost modelling: Before/after analysis — quantify savings from scale-away vs. always-on baseline
• KEDA/Karpenter configuration: Event-driven scaling with custom metrics, cron schedules, and multi-trigger logic
• SLA design: Tiered SLAs that balance cost efficiency with response time guarantees

Every idle GPU is burning money. Scale-away stops the burn. Let's architect your savings.