Kubernetes Secrets Management — External Secrets Operator, Vault, and Sealed Secrets Compared

Introduction

Kubernetes Secrets are not encrypted by default—they're base64-encoded and stored in plain text in etcd. For production workloads, this is unacceptable. You need a secrets management solution that integrates seamlessly with Kubernetes, supports GitOps workflows, enables secret rotation without pod restarts, and provides audit trails. This post compares three production-grade approaches: External Secrets Operator (ESO), HashiCorp Vault with agent injection, and Sealed Secrets.

Why Base64 Is Not Encryption
External Secrets Operator (ESO)
HashiCorp Vault with Agent Injector
Sealed Secrets for GitOps
Secret Rotation Without Pod Restart
RBAC for Secret Access
Auditing Secret Usage
Comparison Summary
Checklist
Conclusion

Why Base64 Is Not Encryption

Kubernetes' default Secret resource is not secure. It's base64-encoded for transport and compatibility, but anyone with etcd access (cluster admins, backup operators) can decode it immediately.

# This "secret" is trivial to extract
kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d

# Better: inspect the raw secret in etcd
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /kubernetes.io/secrets/default/my-secret

For compliance (SOC2, HIPAA, PCI-DSS), you must encrypt secrets at rest and control who accesses them.

External Secrets Operator (ESO)

ESO syncs secrets from external backends (AWS Secrets Manager, Google Secret Manager, HashiCorp Vault) into Kubernetes Secrets. The actual secret data never exists in your Git repository.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: app-credentials
    creationPolicy: Owner
  data:
  - secretKey: db-password
    remoteRef:
      key: prod/database/master-password
  - secretKey: api-key
    remoteRef:
      key: prod/external-api/key

ESO periodically syncs the external secret into a Kubernetes Secret resource. Pods reference the Kubernetes Secret as usual.

apiVersion: v1
kind: Pod
metadata:
  name: app-with-secrets
spec:
  containers:
  - name: app
    image: my-app:v1.2.3
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-credentials
          key: db-password
    - name: API_KEY
      valueFrom:
        secretKeyRef:
          name: app-credentials
          key: api-key

Pros: Centralized secret management in AWS/GCP; secrets in Git never exposed; RBAC on external system. Cons: Another component to operate; sync latency (up to 1 hour); external system outage blocks secret updates.

HashiCorp Vault with Agent Injector

Vault is a purpose-built secrets management system. The Vault Agent Injector is a Kubernetes webhook that injects secrets directly into pods without storing them in etcd.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
  namespace: production
---
apiVersion: vault.hashicorp.com/v1
kind: VaultAuth
metadata:
  name: app-auth
  namespace: production
spec:
  method: kubernetes
  kubernetes:
    role: app-role
    serviceAccount: app-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-db: "secret/data/prod/database"
        vault.hashicorp.com/agent-inject-template-db: |
          {{- with secret "secret/data/prod/database" -}}
          export DB_USER="{{ .Data.data.username }}"
          export DB_PASSWORD="{{ .Data.data.password }}"
          {{- end }}
        vault.hashicorp.com/agent-cache-enable: "true"
        vault.hashicorp.com/agent-limits-cpu: "250m"
        vault.hashicorp.com/agent-limits-mem: "128Mi"
    spec:
      serviceAccountName: app-sa
      containers:
      - name: api
        image: my-app:v1.2.3
        command:
        - /bin/sh
        - -c
        - |
          source /vault/secrets/db
          exec /app/start.sh
        resources:
          requests:
            cpu: "500m"
            memory: "256Mi"
          limits:
            cpu: "1000m"
            memory: "512Mi"

Vault Agent injects secrets as files or environment variables. The webhook intercepts pod creation and adds the agent sidecar.

Pros: No secrets in etcd; agent handles authentication; Vault native audit logs; fine-grained policies; dynamic secrets (databases, SSH keys). Cons: Vault cluster is critical infrastructure; higher operational complexity; sidecar adds latency and resource overhead.

Sealed Secrets for GitOps

Sealed Secrets encrypt secrets with a public key, allowing encrypted data to be stored in Git. Only the cluster (with the private key) can decrypt them.

# Install sealed-secrets
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

# Encrypt a secret
echo -n mysecretpassword | kubectl create secret generic db-pass \
  --dry-run=client \
  --from-file=password=/dev/stdin \
  -o yaml | kubeseal -f - > db-pass-sealed.yaml

# db-pass-sealed.yaml is safe to commit to Git

The sealed secret manifest:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-pass
  namespace: production
spec:
  encryptedData:
    password: AgBvJ5K1xH8Z7n... # Encrypted with cluster public key
  template:
    metadata:
      name: db-pass
      namespace: production
    type: Opaque

When applied, the Sealed Secrets controller decrypts and creates a Kubernetes Secret.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: database-migration
spec:
  template:
    spec:
      containers:
      - name: migrator
        image: postgres-migrator:v1
        env:
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: db-pass
              key: password
        command:
        - /bin/sh
        - -c
        - psql -h db.example.com -U postgres < schema.sql

Pros: Secrets in Git (encrypted); no external system dependency; simple to operate; GitOps-friendly. Cons: Decryption key on cluster (risk if cluster compromised); no rotation mechanism; less fine-grained audit.

Secret Rotation Without Pod Restart

Kubernetes doesn't automatically restart pods when secrets change. You need a reload strategy.

Option 1: Reloader (watches Secret changes, restarts pods)

helm repo add stakater https://stakater.github.io/stakater-helm-charts
helm install reloader stakater/reloader -n kube-system

Annotate your pod:

spec:
  template:
    metadata:
      annotations:
        reloader.stakater.com/match: "true"

When a Secret changes, Reloader patches the pod's annotations, triggering a rolling restart.

Option 2: Vault agent template rendering

Vault Agent periodically re-renders secret files. Apps watch file changes:

vault.hashicorp.com/agent-inject-file-db: "db/config.sh"
vault.hashicorp.com/agent-inject-template-db: |
  {{- with secret "secret/data/prod/database" -}}
  DB_PASSWORD="{{ .Data.data.password }}"
  {{- end }}

Your app watches /vault/secrets/db/config.sh and reloads when it changes.

Option 3: Dynamic credentials (Vault databases)

For databases, use Vault's database secret engine to generate ephemeral credentials:

vault.hashicorp.com/agent-inject-secret-db: "database/static-creds/postgres-app"

Vault automatically rotates the underlying database password; your app gets fresh credentials on each injection.

RBAC for Secret Access

Prevent developers from reading secrets meant for other teams. Use RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: api-secrets-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["api-credentials", "api-tls-cert"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: api-secrets-reader-binding
  namespace: production
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: api-secrets-reader
subjects:
- kind: ServiceAccount
  name: api-app-sa
  namespace: production

This allows the api-app-sa service account to read only api-credentials and api-tls-cert, no others.

For Vault RBAC:

path "secret/data/prod/api/*" {
  capabilities = ["read", "list"]
}

path "secret/data/prod/database/*" {
  capabilities = []  # Deny
}

Auditing Secret Usage

Kubernetes audit logs: Enable audit logging and monitor Secret access:

kind: AuditPolicy
apiVersion: audit.k8s.io/v1
rules:
- level: RequestResponse
  verbs: ["get", "watch", "list"]
  resources:
  - group: ""
    resources: ["secrets"]
  omitStages:
  - RequestReceived

Vault audit logs: Vault logs all secret reads to file or syslog:

vault audit list
# Output:
# Path     Type       Description
# ----     ----       -----------
# file/    file       standard output
# syslog/  syslog     syslog

Parse and alert on unexpected access patterns.

Comparison Summary

Feature	ESO	Vault	Sealed Secrets
Secrets in Git	No	No	Yes (encrypted)
External system required	Yes	Yes	No
Dynamic credentials	No	Yes	No
Rotation without restart	Via Reloader	Native	Via Reloader
Audit trail	Ext system	Excellent	Weak
Operational complexity	Medium	High	Low

Checklist

Conclusion

Storing secrets in plain text Kubernetes Secrets is a risk you can eliminate with modest operational effort. Choose ESO for cloud-native deployments with external secret backends, Vault for sophisticated access control and dynamic credentials, or Sealed Secrets for simplicity in GitOps pipelines. Whichever path you choose, invest in audit logging, RBAC, and rotation automation—your security team (and your customers) will thank you.