Environment-Specific Configuration in Kubernetes: Best Practices

Previously in this course, we explored deployments-declarative-updates-for-kubernetes-applications to ensure our apps stay running through updates. Now that you can deploy, it is time to address a critical operational reality: your application likely needs different settings—database URLs, resource limits, or feature flags—depending on whether it is running in development, staging, or production.

Hardcoding these values into a single YAML file is a recipe for disaster. If you accidentally point your production app to a development database because of a copy-paste error, the consequences are immediate and severe.

The Strategy: Separating Configuration from Code

In a professional environment, we treat our Kubernetes manifests as code. To manage environment configuration effectively, we follow the principle of separation of concerns. You should have a base definition that describes the "what" (your Deployment and Service) and environment-specific overlays that describe the "where" (the specific context).

While sophisticated tools like Helm or Kustomize exist for this purpose, we will focus on the fundamental directory-based approach first. This ensures you understand the underlying structure before adding layers of abstraction.

A Robust Directory Structure

Organize your repository to clearly distinguish between shared resources and environment-specific overrides. A standard layout looks like this:

TEXT
/k8s
  /base
    deployment.yaml
    service.yaml
  /overlays
    /dev
      configmap.yaml
    /staging
      configmap.yaml
    /prod
      configmap.yaml

The /base directory contains your "source of truth." These files should not contain sensitive or environment-specific values. Instead, they reference placeholders or rely on managing-application-configuration-with-kubernetes-configmaps to inject values at runtime.

Worked Example: The Pattern

Suppose you have a web application. In base/deployment.yaml, you define the image and the container port, but you leave the environment variables to be injected via a ConfigMap.

YAML
# base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-web-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: my-repo/web-app:v1
        envFrom:
        - configMapRef:
            name: app-config # The name is consistent, the content changes

Now, in your /overlays/dev/configmap.yaml, you define the connection strings for your local database:

YAML
# overlays/dev/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_URL: "localhost:5432"
  LOG_LEVEL: "debug"

In /overlays/prod/configmap.yaml, you define the production connection strings:

YAML
# overlays/prod/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DB_URL: "prod-db-cluster:5432"
  LOG_LEVEL: "warn"

Hands-on Exercise: Implementing Environment Overrides

1. Create a base folder and move your existing application manifest into it.
2. Remove any hardcoded environment-specific values (like database connection strings or replicas) from the base manifest.
3. Create an overlays folder. Inside, create a dev folder and a prod folder.
4. Create a configmap.yaml in each folder with the appropriate values for that environment.
5. Apply the base config first: kubectl apply -f base/.
6. Apply the environment-specific config: kubectl apply -f overlays/dev/.

Verify the injection by running kubectl describe pod <pod-name> and checking the Environment section.

Common Pitfalls in Configuration Management

• Drift: When your base manifests change but your overlays remain outdated. Always treat your overlays as first-class citizens in your CI/CD pipeline.
• Secret Sprawl: Never store production database passwords in plain text ConfigMaps. As we covered in using-kubernetes-secrets-a-practical-guide-to-sensitive-data, use Secrets for sensitive data and keep them out of your git history.
• Over-templating: Don't build a complex templating engine if you only have two environments. Start simple with directory separation; move to tools like Helm only when you reach a "breaking point" where manual updates become error-prone.

Frequently Asked Questions

Q: Should I use namespaces or separate clusters for environments? A: Use namespaces for development and staging to save costs. For production, a dedicated, isolated cluster is the industry standard for security and blast-radius containment.

Q: How do I know which environment I am currently pointing to? A: Use kubectl config current-context to verify your target cluster. Always double-check your context before running kubectl apply.

Q: Can I use one file for everything? A: You can, but you shouldn't. It violates the DRY (Don't Repeat Yourself) principle and makes it impossible to apply changes to one environment without risking the others.

Recap

Managing environment-specific configuration is about discipline. By separating your base application logic from the environment-specific data, you create a system that is predictable and safe. Remember: base manifests define the "what," and overlays define the "where."

Up next: We will look at Advanced Labeling Strategies to help you manage these multi-environment resources more granularly.