Kubernetes Secret Management: Vault and External Secrets Operator

We’ve all been there. You’re deploying a new microservice, and you need to inject a database password or an API key. The "junior" way is putting it in a plain Kubernetes Secret YAML in your Git repo. Don’t do that. Even if it’s base64 encoded, it’s plain text to anyone with read access to your repository.

In this post, I’ll show you how to implement professional-grade Kubernetes secret management using HashiCorp Vault and the External Secrets Operator (ESO). This setup keeps your secrets encrypted at rest in Vault and syncs them into Kubernetes only when needed.

Why use the External Secrets Operator?

Native Kubernetes secrets are notoriously insecure. They lack audit trails, automatic rotation, and centralized policy management. By using the External Secrets Operator (v0.9.x+), you treat Vault as the "source of truth." ESO acts as a bridge: it fetches secrets from Vault and creates native Kubernetes secrets for your pods to consume.

It’s the cleanest way to maintain a GitOps workflow without exposing sensitive data in your version control.

Prerequisites

Before we jump into the code, make sure you have:

1. A running Kubernetes cluster (v1.25+ recommended).
2. A HashiCorp Vault instance (v1.14+).
3. Helm installed for managing package deployments.

Step 1: Install External Secrets Operator

First, let's get ESO running in your cluster. I prefer using Helm for this because it makes upgrades painless.

Bash
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
    -n external-secrets --create-namespace

Verify it’s running with kubectl get pods -n external-secrets. You should see the controller and the webhook pod up and healthy.

Step 2: Configure Vault for Kubernetes Auth

You don't want to use root tokens. Instead, configure Vault to authenticate your Kubernetes pods using their ServiceAccount tokens.

Enable the Kubernetes auth method in Vault:

Bash
vault auth enable kubernetes

vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Create a Vault policy that grants read access to your secrets path:

HCL
# policy.hcl
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

Step 3: Define the SecretStore

The SecretStore resource tells ESO how to talk to Vault. It’s namespaced, meaning you can have different stores for different environments (dev, staging, prod).

YAML
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-role"

Step 4: Map External Secrets to Kubernetes

Now, define an ExternalSecret object. This is the resource that actually pulls the data from Vault and creates a standard Secret in your namespace.

YAML
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-db-credentials
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials-secret # The resulting K8s secret
  data:
  - secretKey: password
    remoteRef:
      key: secret/myapp/db
      property: password

Once you apply this, ESO reaches out to Vault, retrieves the value, and creates a Kubernetes Secret named db-credentials-secret. Your application pods can now mount this secret as environment variables or files just like any other native secret.

Lessons from Production

I’ve managed this in high-traffic environments, and here are three things I’ve learned the hard way:

1. Namespace Isolation: Use one SecretStore per namespace to keep blast radii small. Don't share a global store unless you have strict reasons.
2. Refresh Intervals: Setting refreshInterval to something like 1h is fine for most apps. If you need immediate rotation, trigger a rollout restart via your CI/CD pipeline after updating Vault.
3. Monitoring: Always monitor the ESO logs. If the operator loses connection to Vault, your pods will fail to start when they try to mount missing secrets. Use Prometheus to alert on external_secrets_sync_error metrics.

Final Thoughts

Implementing cloud-native security isn't just about picking the right tools; it’s about removing the "human element" from secret handling. By automating the sync between Vault and Kubernetes, you eliminate the risk of developers accidentally committing secrets to Git.

This is the standard for modern GitOps secrets management. It’s secure, scalable, and keeps your audit logs clean. If you're still relying on manual kubectl create secret commands, it’s time to move on.