Using Kubernetes Secrets: A Practical Guide to Sensitive Data

Previously in this course, we discussed managing application configuration with kubernetes configmaps, which is perfect for non-sensitive data like feature flags or UI settings. However, when you need to handle sensitive data like database credentials or API keys, you must use Secrets.

Managing sensitive data requires a shift in mindset. You shouldn't store secrets in your source control, and you shouldn't pass them as plaintext in your environment variables if they can be avoided. In Kubernetes, the Secret object is designed specifically to decouple your application code from its credentials.

Understanding Encoding vs. Encryption

A common point of confusion for beginners is the security level of a Kubernetes Secret. By default, Kubernetes stores secrets in etcd (the cluster state store we covered in the-control-plane-and-the-state-store-kubernetes-architecture) as base64 encoded strings.

• Encoding is NOT encryption. Base64 is a reversible format, not a security mechanism. Anyone with access to the Kubernetes API can decode your secrets.
• Encryption at Rest: To truly secure secrets, you must enable encryption at rest for your etcd database. This is a cluster-level configuration that ensures even if someone gets a raw dump of the database file, they cannot read the values without the decryption key.

Always treat base64-encoded manifests as plaintext. Never commit them to Git.

Creating a Secret

You can create a secret imperatively using kubectl or declaratively via a YAML manifest. For most production workflows, use a manifest, but let's look at the manual way to understand the structure.

To create a secret for a database password:

Bash
# Create a secret named 'db-secret' containing a username and password
kubectl create secret generic db-secret \
  --from-literal=username=admin \
  --from-literal=password=supersecret123

If you inspect this resource with kubectl get secret db-secret -o yaml, you will see your keys, but the values will be in that base64 format we discussed.

Injecting Secrets into Containers

Once created, you need to make these secrets available to your application. The most common method is injecting them as environment variables, similar to how we handled standard configuration in injecting-environment-variables-in-kubernetes-pods.

Update your Pod manifest to reference the secret:

YAML
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  containers:
  - name: my-app
    image: nginx
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-secret
          key: password

When the Pod starts, Kubernetes pulls the values from db-secret, decodes them, and injects them directly into the container's environment.

Hands-on Exercise

1. Create: Use the kubectl create secret command above to generate your own db-secret.
2. Verify: Run kubectl get secret db-secret -o yaml and notice how the values look. Try decoding one of the strings using echo "value" | base64 --decode in your terminal to see the original text.
3. Deploy: Create a Pod using the YAML provided above.
4. Inspect: Once running, verify the variables exist inside the container: kubectl exec secure-app -- printenv | grep DB_

Common Pitfalls

• Committing Secrets to Git: This is the #1 security failure. Even if you delete the secret later, your Git history will contain the sensitive data forever. Use tools like git-secrets or trufflehog to scan your commits.
• Assuming Base64 is Secure: Never assume that putting a value into a Kubernetes Secret makes it "hidden." It only makes it "managed."
• Logging Secrets: If your application logs its environment variables on startup (a common pattern in many frameworks), your secrets will end up in your logs. Always sanitize your startup logging.
• Permissions: Secrets are namespaced. If you are struggling to access a secret, ensure your Pod is in the same namespace as the secret object.

FAQ

Q: Can I update a secret without restarting the Pod? A: If you inject secrets as environment variables, the Pod will not see changes until it is restarted. If you mount them as files (volumes), updates can be detected, though this depends on how your app reads the file.

Q: Are there alternatives to Kubernetes Secrets? A: Yes. Many production environments use external providers like HashiCorp Vault or cloud-native solutions like AWS Secrets Manager to inject secrets into pods via sidecars or CSI drivers.

Recap

Secrets are the standard way to handle sensitive data in Kubernetes. By referencing secretKeyRef in your pod manifests, you avoid hardcoding credentials. Remember: encoding is not encryption, and protecting your secrets in source control is a non-negotiable security practice.

Up next: We will look at managing storage, specifically how to move beyond ephemeral data with Persistent Volumes.