Luca BertonThe Kubernetes Secret Problem
Kubernetes Secrets are base64-encoded, not encrypted. Anyone with RBAC access to a namespace can read every secret in it. For regulated enterprises, storing sensitive credentials directly in Kubernetes is a compliance risk. The solution: store secrets in a dedicated secrets manager (Vault, AWS Secrets Manager, Azure Key Vault) and synchronise them into Kubernetes on demand.
External Secrets Operator (ESO) is the standard Kubernetes operator for this pattern, supported by the CNCF ecosystem.
How ESO Works
Architecture
- You define a SecretStore — connection to your secrets backend (Vault, AWS SM, Azure KV, etc.)
- You define an ExternalSecret — which secret to fetch, how to map it, and refresh interval
- ESO fetches the secret from the external backend and creates a Kubernetes Secret
- ESO continuously reconciles — when the external secret changes, the Kubernetes Secret is updated
- Pods reference the Kubernetes Secret as normal (env vars, volume mounts)
Key benefit: Developers use standard Kubernetes Secret references. They don't need to know which backend the secret comes from or how authentication works.
Supported Backends
- HashiCorp Vault: KV v1/v2, dynamic secrets, PKI
- AWS Secrets Manager: Full integration with IAM authentication
- AWS Parameter Store: Cost-effective alternative for simpler secrets
- Azure Key Vault: Secrets, keys, and certificates
- GCP Secret Manager: Full integration with Workload Identity
- 1Password, Doppler, Infisical: Developer-friendly backends
- Oracle Vault, IBM Secrets Manager: Enterprise backends
Compliance Benefits
- Centralised audit: All secret access logged in the secrets backend (Vault audit log, AWS CloudTrail, Azure Activity Log)
- No secrets in Git: GitOps repos contain ExternalSecret references, not actual secret values
- Rotation support: When secrets rotate in the backend, ESO automatically updates Kubernetes Secrets
- RBAC separation: Platform team manages SecretStores, development teams manage ExternalSecrets — separation of duties
- Multi-tenancy: ClusterSecretStore for shared backends, namespace-scoped SecretStore for team-specific backends
Kubernetes Recipes
A practical guide for container orchestration and deployment by Grzegorz Stencel & Luca Berton (Apress).
Watch on Skillshare →ESO vs Vault Agent vs CSI Driver
| Approach | ESO | Vault Agent | Secrets Store CSI |
|---|---|---|---|
| Backend agnostic | Yes (20+ backends) | Vault only | Provider-specific |
| Sidecar needed | No (operator-based) | Yes (sidecar per pod) | No (CSI volume) |
| Creates K8s Secret | Yes (standard Secret) | Files in pod (not a K8s Secret) | Optional |
| Resource overhead | One operator for all secrets | One sidecar per pod using secrets | One DaemonSet |
Implementation Guide
- Install ESO via Helm in your cluster
- Create a ClusterSecretStore pointing to your primary secrets backend
- Migrate one service — replace hardcoded secrets with ExternalSecret references
- Set refresh interval — balance between freshness and API rate limits (5-15 minutes typical)
- Monitor with Prometheus — ESO exposes metrics for sync status, errors, and latency
Back-End Infrastructure: Servers, Secure APIs and Data
Build secure back-end infrastructure from the ground up. In collaboration with Starweaver.
Start on Coursera →