Kubernetes Secret Management: Using External Secrets and HashiCorp Vault

Why Native Kubernetes Secrets Fail at Scale

I’ve spent years cleaning up messes caused by base64 encoded secrets stored directly in Git. If you're still committing manifests with stringData to your repository, stop. It’s a ticking time bomb. Kubernetes native secrets aren't encrypted at rest by default, and they lack the audit trails required for enterprise compliance.

To move toward production-grade Kubernetes Secret Management, you need a centralized source of truth. For most of my projects, that's HashiCorp Vault. But mounting Vault directly into pods via sidecars is clunky. That’s where the External Secrets Operator (ESO) changes the game. It bridges the gap by fetching secrets from your vault and syncing them into native Kubernetes Secret objects.

The Architecture: How ESO and Vault Work Together

The External Secrets Operator operates on a simple CRD-based loop. You define an ExternalSecret resource, which tells the operator: "Go to Vault, grab this specific key, and turn it into a native K8s Secret."

Here’s the flow:

1. Vault: Holds the actual sensitive data (e.g., API keys, DB credentials).
2. ESO: Watches for ExternalSecret resources in your cluster.
3. SecretStore: Defines the connection parameters (URL, auth method) to Vault.
4. Kubernetes Secret: The final object created by ESO, which your application consumes like any other secret.

Step 1: Installing the External Secrets Operator

I prefer installing via Helm. It’s clean, versioned, and easy to roll back.

Bash
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets --create-namespace \
    --set installCRDs=true

I’m currently using version 0.9.18. Always check the official docs for breaking changes before upgrading.

Step 2: Configuring the SecretStore

The SecretStore is the bridge. Since I usually run Vault in the same cluster (or a peered VPC), I use the Kubernetes Auth method for authentication.

YAML
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: app-prod
spec:
  provider:
    vault:
      server: "https://vault.internal.mycompany.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-role"

Pro-tip: Ensure the eso-role in Vault has a policy that limits access to only the specific paths your application needs. Never use a global read token.

Step 3: Implementing Secret Injection

Now, define the ExternalSecret resource in your GitOps pipeline. This is where GitOps Security meets Secrets Injection. You aren't storing the secret; you're storing the reference to the secret.

YAML
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: app-prod
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-creds-secret # The K8s secret to be created
  data:
  - secretKey: username
    remoteRef:
      key: database/config/app
      property: username

Once you apply this, ESO immediately polls Vault. If the secret exists, you’ll see the db-creds-secret object appear in your namespace. Your deployment just references this like it would any standard secret.

Handling Rotation and Syncing

The refreshInterval is your best friend. In the example above, I set it to 1h. If you update the password in Vault, ESO will automatically update the Kubernetes secret within the hour without requiring a pod restart (unless your app reads the secret at startup).

If your app does need a restart on secret change, use the Reloader tool. It watches for changes in ConfigMaps and Secrets and triggers a rolling update of your deployments.

Why this wins for GitOps

Using External Secrets Operator with HashiCorp Vault allows you to keep your Git repositories clean. Your manifests become environment-agnostic. You don't need to change your application code or your CI/CD pipeline when a secret changes; you just update the value in Vault.

I’ve seen teams try to build custom controllers for this. Don't. ESO is battle-tested, handles rate-limiting, and supports multiple backends (AWS Secrets Manager, GCP Secret Manager, etc.). By offloading the complexity of Secrets Injection to a mature operator, you can focus on building features instead of debugging secret propagation issues.

Keep your secrets out of Git, keep your Vault audit logs tight, and let the operator do the heavy lifting.