Rolling Back Deployments: A Guide to Kubernetes Recovery

Previously in this course, we explored Deployments: Declarative Updates for Kubernetes Applications, where we learned how to perform rolling updates to transition your application from one version to the next without downtime. In this lesson, we add the "safety net": how to handle the inevitable scenario where that new version introduces a bug or performance regression.

The Necessity of Version Control and Recovery

When you update a Deployment, Kubernetes doesn't just overwrite the old configuration; it creates a new "Revision." This is the core of the Kubernetes Rollback mechanism. By treating your deployment history as a series of snapshots, Kubernetes allows you to revert to a known good state instantly. This is a critical skill for any engineer, as it minimizes the "Mean Time to Recovery" (MTTR) when production environments face issues.

Viewing Your Deployment History

Before you can undo a change, you need to see what has happened. Every time you update a Deployment's image or configuration, Kubernetes stores the state of that change.

To see the revision history of your deployment, use:

Bash
kubectl rollout history deployment/<deployment-name>

If you want to see the specific details of a particular revision (like which container image was used), you can append the revision number:

Bash
kubectl rollout history deployment/<deployment-name> --revision=2

Performing a Rollback

If you discover that your latest deployment is causing errors, you don't need to manually revert your YAML files or re-deploy the old image. You can tell Kubernetes to return to the previous state automatically.

To undo the most recent change:

Bash
kubectl rollout undo deployment/<deployment-name>

If you need to jump back to a specific, older revision rather than just the immediate previous one, specify it explicitly:

Bash
kubectl rollout undo deployment/<deployment-name> --to-revision=1

Once you run this command, Kubernetes initiates a new rollout process. The Deployment controller will terminate the current (faulty) Pods and replace them with the Pods defined in the target revision.

Hands-on Exercise: The Broken Update

Let’s simulate a failure and perform a recovery.

1. Deploy a stable version: Update your deployment to use nginx:1.21. kubectl set image deployment/web-app nginx=nginx:1.21

2. Trigger a "bad" update: Let's simulate a broken release by pointing to a non-existent tag. kubectl set image deployment/web-app nginx=nginx:999.999

3. Check the status: Run kubectl get pods. You will see ImagePullBackOff or ErrImagePull.

4. Roll back to safety: Run kubectl rollout undo deployment/web-app.

5. Verify: Watch the pods return to the Running state with the 1.21 image.

Common Pitfalls

• Assuming Rollbacks are Instant: Even though the command is fast, the actual transition follows your strategy (e.g., RollingUpdate). It still takes time to pull images and pass readiness probes.
• Forgetting to update the source YAML: A rollback only updates the live state in the cluster. If you don't update your local Git repository or source YAML files to match the rolled-back version, the next time you run kubectl apply, you will redeploy the broken configuration.
• Too many revisions: By default, Kubernetes keeps a limited number of old ReplicaSets to track history. If you perform dozens of updates, you might lose the ability to go back to very old versions.

FAQ

Q: Does a rollback work if I haven't used kubectl set image? A: Yes. kubectl rollout tracks changes to the Pod template in your deployment. If you updated the manifest via kubectl apply, that also creates a new revision.

Q: Can I pause a rollout if I see it failing? A: You can use kubectl rollout pause deployment/<name> to stop a rolling update in progress, though this is usually for debugging or configuration changes mid-update.

Recap

We have learned that Deployment recovery in Kubernetes is a built-in feature, not an afterthought. By utilizing kubectl rollout history to track changes and kubectl rollout undo to revert to stable snapshots, you ensure your infrastructure remains resilient. Remember: always sync your source control after performing a manual rollback to prevent "drift" between your cluster state and your configuration files.

Up next: Scaling Applications, where we will learn how to adjust the number of running Pods on the fly using kubectl scale.