Modifying Running Pods: Patching, Updating, and Recreating

Previously in this course, we covered injecting environment variables in Kubernetes pods to configure our applications dynamically. Now that you have configured apps running in your cluster, you will inevitably need to change them—whether to update an image version, adjust labels, or tweak resource settings.

In Kubernetes, you don't "edit" a running Pod in the same way you might edit a file on a server. Instead, you manage the desired state of your cluster.

The Declarative Update Workflow

Kubernetes is designed around the declarative model we discussed in our lesson on declarative vs. imperative models in Kubernetes. You maintain a source-of-truth YAML file, and the cluster reconciles the difference between that file and reality.

When you need to modify a running Pod, you follow this cycle:

1. Update your local YAML manifest.
2. Run kubectl apply -f your-pod.yaml.
3. Kubernetes compares the new manifest against the existing resource.

Immutable vs. Mutable Fields

Not all changes are created equal. When you apply a change, the Kubernetes API server determines if the update is allowed based on the field you are modifying.

• Mutable Fields: Fields like labels, annotations, or image (under specific conditions) can often be updated without terminating the Pod.
• Immutable Fields: Many core configuration fields, such as the command used to start the container, volume mounts, or the uid of the resource, are immutable. If you change these, the API server will reject the apply command with an error stating that the field cannot be updated.

Worked Example: Updating a Running Pod

Let’s perform a live update. Assume you have a running Pod defined in web-app.yaml.

YAML
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: web
spec:
  containers:
  - name: nginx
    image: nginx:1.21

Suppose you need to update the label to app: web-server and change the image to nginx:1.25.

Step 1: Modify the Manifest Open web-app.yaml and update the values:

YAML
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    app: web-server  # Changed
spec:
  containers:
  - name: nginx
    image: nginx:1.25  # Changed

Step 2: Apply the Change Run the following command: kubectl apply -f web-app.yaml

If the change is allowed, you will see pod/nginx-pod configured. If you try to change an immutable field, you will receive an error.

Why Recreate?

When a change is rejected because the field is immutable, you must recreate the Pod. This is a deliberate design choice in Kubernetes to ensure that the container runtime environment remains consistent throughout the Pod's lifecycle.

To recreate the Pod:

1. Delete the existing Pod: kubectl delete pod nginx-pod.
2. Apply the new manifest: kubectl apply -f web-app.yaml.

In production environments, we rarely manage individual Pods this way because deleting a Pod causes downtime. We use Deployments (which we will cover later in the course) to perform "rolling updates" that replace Pods automatically without service interruption.

Hands-on Exercise

1. Create a file named debug-pod.yaml with a simple BusyBox image.
2. Apply it to your cluster.
3. Attempt to add a new label to the metadata section of the YAML file and run kubectl apply. Verify the change with kubectl get pod --show-labels.
4. Now, try to change the command field of the container in the same YAML file. Observe the error message returned by kubectl.

Common Pitfalls

• Forgetting to save the file: kubectl apply reads the file on your disk. If you forget to save your editor before running the command, you are applying the old configuration.
• Assuming Live Updates are Safe: Just because you can update an image tag doesn't mean it's safe. If the new image crashes, your Pod will enter a CrashLoopBackOff state immediately.
• Mixing Imperative and Declarative: If you use kubectl edit (imperative) to change a Pod, your local YAML file is now out of sync with the cluster. Always update your source-of-truth file first.

FAQ

Q: Can I change the container port of a running Pod? A: No, the ports specification is immutable. You must delete and recreate the Pod to change it.

Q: Does kubectl apply restart the container? A: Only if you change a field that triggers a container restart (like the image). Simple label changes do not restart the container.

Q: How do I know which fields are immutable? A: The API server will explicitly tell you in the error message if you try to change one. Generally, anything that affects the container's runtime environment or the Pod's identity is immutable.

Recap

We’ve learned that while Kubernetes allows for some live updates (like labels), most fundamental configuration changes require the Pod to be recreated. Always prefer declarative kubectl apply over imperative edit commands to ensure your local files remain the source of truth.

Up next: We will learn how to safely perform these operations by cleaning up resources and managing the lifecycle of your manifests.