Kubernetes Secrets Management with HashiCorp Vault

Managing secrets in a Kubernetes environment can feel like walking a tightrope. You need to keep your sensitive data — API keys, database credentials, TLS certificates — safe, but also make them accessible to your applications. Traditional methods, like storing secrets directly in Kubernetes Secrets objects, often fall short, especially as your infrastructure scales and your security requirements grow. That's where HashiCorp Vault comes in.

In this post, I'll walk you through how I've implemented a secure and scalable secrets management strategy using HashiCorp Vault in a Kubernetes cluster. We'll cover the core concepts, the setup process, and how to leverage Vault's Kubernetes Auth method to securely inject secrets into your pods.

Why HashiCorp Vault for Kubernetes Secrets?

Kubernetes Secrets are a good starting point, but they have limitations:

• Limited Auditing: While Kubernetes logs Secret access, it's not as granular or centralized as Vault's audit logs.
• Static Secrets: Secrets are stored statically. If a compromise occurs, you need to manually rotate them across all Secrets objects.
• Encryption at Rest: Kubernetes Secrets are only encrypted at rest if the etcd datastore is configured for encryption. This is often an afterthought.
• Dynamic Secrets: Vault can generate dynamic, short-lived credentials on demand, significantly reducing the attack surface.

HashiCorp Vault addresses these shortcomings by providing a centralized, secure, and feature-rich platform for managing secrets. It offers:

• Centralized Management: One place to manage all your secrets.
• Dynamic Secrets: Generate credentials that expire automatically.
• Leasing and Renewal: Secrets are leased and can be renewed, not permanent.
• Detailed Auditing: Comprehensive audit logs of who accessed what, when.
• Encryption: Strong encryption for secrets at rest and in transit.
• Pluggable Secrets Engines: Support for various secret types (databases, cloud providers, PKI, etc.).

Setting Up HashiCorp Vault in Kubernetes

There are a few ways to run Vault in Kubernetes:

1. Standalone: Running Vault as a single instance. Not recommended for production due to lack of high availability.
2. High Availability (HA): Running multiple Vault instances behind a load balancer. This is the standard for production.

For production, I strongly recommend the HA setup. HashiCorp provides an excellent Helm chart for deploying Vault in HA mode.

Prerequisites

• A Kubernetes cluster (e.g., GKE, EKS, AKS, or self-hosted).
• kubectl configured to communicate with your cluster.
• Helm v3 installed.

Deploying Vault with Helm

First, add the HashiCorp Helm repository:

Bash
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Next, create a values.yaml file for your Vault deployment. This is where you'll configure Vault for HA, storage, and other settings.

YAML
# values.yaml
server:
  # Number of Vault instances for HA
  ha:
    enabled: true
    replicas: 3

  # Configure storage backend. For production, use a reliable,
  # shared backend like Consul, etcd, or an external database.
  # This example uses the default `inmem` for simplicity during testing,
  # but you'll want to change this for production.
  #
  # For production, consider Consul:
  # backend: "consul"
  # consul:
  #   address: "consul.service.consul:8500" # Replace with your Consul service address
  #   enabled: true
  #
  # Or etcd:
  # backend: "etcd"
  # etcd:
  #   enabled: true
  #   address: "http://etcd.service.etcd:2379" # Replace with your etcd service address
  #   haEnabled: true
  #   peerURLs: ["http://vault-0.vault-internal.default.svc:8201", "http://vault-1.vault-internal.default.svc:8201", "http://vault-2.vault-internal.default.svc:8201"] # Adjust as needed

  # Enable the Kubernetes auth method
  kubernetes:
    enabled: true

  # Configure audit logging
  audit:
    enabled: true
    logPath: "/vault/logs/audit.log"
    # You can configure outputs like syslog or file
    # output: "file"

# Ensure you have a Kubernetes ServiceAccount for Vault
ui:
  enabled: true
  serviceType: "ClusterIP"
  # Set this to your Kubernetes namespace where Vault will be deployed
  # namespaceOverride: "vault"

# If you are using a specific namespace for Vault
# namespace: vault

Now, deploy Vault using Helm:

Bash
# Create a namespace for Vault if it doesn't exist
kubectl create namespace vault

# Deploy Vault
helm install vault hashicorp/vault \
  --namespace vault \
  -f values.yaml

After deployment, you'll need to initialize Vault and unseal it. The first time, you'll need to unseal it manually. Subsequent unseals can be automated using auto-unseal features (e.g., with KMS or a Shamir secret sharing auto-unseal key).

Initialize and Unseal (First Time)

1. Get the Vault Pod Name:

   Bash
   kubectl get pods -n vault -l app=vault -o jsonpath='{.items[0].metadata.name}'
   

2. Initialize Vault:

   Bash
   kubectl exec <vault-pod-name> -n vault -- vault operator init -key-shares=5 -key-threshold=3
   

   This will output several unseal keys and a root token. Store these securely! You'll need 3 out of 5 keys to unseal.
3. Unseal Vault:

   Bash
   kubectl exec <vault-pod-name> -n vault -- vault operator unseal <key1>
   kubectl exec <vault-pod-name> -n vault -- vault operator unseal <key2>
   kubectl exec <vault-pod-name> -n vault -- vault operator unseal <key3>
   

4. Login with Root Token:

   Bash
   # Get the root token from the init output
   export VAULT_TOKEN="<your-root-token>"
   kubectl exec <vault-pod-name> -n vault -- vault status
   

Configuring Vault for Kubernetes Auth

Once Vault is running and accessible, you need to configure the Kubernetes authentication method. This allows pods to authenticate with Vault using their Kubernetes Service Account tokens.

1. Enable the Kubernetes Auth Method:

   Bash
   vault auth enable kubernetes
   

2. Configure the Kubernetes Auth Method: You need to tell Vault how to communicate with your Kubernetes API. This involves providing the Kubernetes CA certificate and the host URL. Vault can usually discover this automatically if it's running within the cluster.

   Bash
   # Get the Kubernetes API server address and CA certificate
   KUBE_HOST=$(kubectl config view --raw -o jsonpath='{.clusters[].cluster.server}')
   KUBE_CA_CERT=$(kubectl config view --raw -o jsonpath='{.clusters[].cluster.certificate-authority-data}')
   
   vault write auth/kubernetes/config \
       token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
       kubernetes_host="${KUBE_HOST}" \
       kubernetes_ca_cert="${KUBE_CA_CERT}" \
       issuer="https://kubernetes.default.svc.cluster.local"
   

   Note: The token_reviewer_jwt is automatically mounted inside the Vault pods.

3. Create a Vault Role: A Vault role defines which Kubernetes Service Accounts are allowed to authenticate and what policies they will have.

   Bash
   # Example: Allow the 'default' service account in the 'default' namespace
   # to authenticate and receive a Vault token with the 'my-app-policy'
   vault write auth/kubernetes/role/my-app-role \
       bound_service_account_names=default \
       bound_service_account_namespaces=default \
       policies=my-app-policy \
       ttl=24h
   

   • bound_service_account_names: The specific Service Account(s) allowed.
   • bound_service_account_namespaces: The namespace(s) where the Service Account must exist.
   • policies: The Vault policies to attach to the generated token.
   • ttl: The Time-To-Live for the Vault token.

4. Create a Vault Policy: Define the permissions your application will have within Vault.

   Bash
   vault policy write my-app-policy - <<EOF
   # Allow reading secrets from a specific path
   path "secret/data/my-app/*" {
     capabilities = ["read"]
   }
   EOF
   

Injecting Secrets into Kubernetes Pods

Now that Vault is set up and configured for Kubernetes auth, let