Cluster Upgrades and Maintenance: How to Safely Drain Nodes

Previously in this course, we explored Multi-Container Pods to handle complex application sidecars. Now that you understand how to structure your workloads, you need to know how to keep the underlying infrastructure healthy through regular maintenance and cluster upgrades.

In a production environment, your nodes—the machines running your pods—require periodic attention, such as kernel updates, security patching, or hardware replacement. You cannot simply shut down a server running your applications without causing service disruption. Kubernetes provides a built-in workflow to handle this gracefully.

The Node Maintenance Workflow

When you need to perform maintenance on a worker node, the objective is to move all running workloads to other healthy nodes in the cluster without dropping connections. This process, often called "draining," involves two primary steps:

1. Cordoning: Marking the node as "unschedulable." This prevents the scheduler from placing any new pods on this node.
2. Draining: Evicting the existing pods from the node. Kubernetes will signal these pods to terminate gracefully, and if they are managed by a controller like a Deployment (as we learned in Deployments: Declarative Updates), they will be automatically recreated on other nodes.

For a deeper look at how these nodes fit into the larger architecture, revisit our lesson on the Anatomy of a Kubernetes Cluster.

How to Use kubectl drain

Before you begin, identify the node you intend to maintain using kubectl get nodes. Once you have the node name, follow this standard procedure.

Step 1: Cordon the node

First, stop new pods from landing on the node:

Bash
kubectl cordon <node-name>

Step 2: Drain the node

Next, force the eviction of existing pods. This command will report any pods that cannot be moved (like those not managed by a controller):

Bash
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

• --ignore-daemonsets: Essential because DaemonSets (like log collectors) are tied to the node; you generally cannot "move" them.
• --delete-emptydir-data: Required if your pods use local ephemeral storage, as Kubernetes needs permission to discard that local data during eviction.

Once the command finishes, the node is ready for maintenance. After your work is complete, bring the node back into service by uncordoning it:

Bash
kubectl uncordon <node-name>

Hands-on Exercise

Using your local cluster, practice the maintenance cycle:

1. List your current nodes and pick one.
2. Run kubectl cordon <node-name>.
3. Observe the status change in kubectl get nodes (it should show SchedulingDisabled).
4. Run the drain command provided above to move your existing pods.
5. Verify the pods have moved to a different node using kubectl get pods -o wide.
6. Return the node to the cluster with kubectl uncordon <node-name>.

Common Pitfalls

• Forgetting Disruption Budgets: If you have configured PodDisruptionBudgets (PDBs), kubectl drain will wait for them to be satisfied. If your PDB is too strict, the drain command may hang indefinitely.
• DaemonSet Errors: If you omit --ignore-daemonsets, the drain will fail. Always verify which resources are running on the node before starting.
• Single-Node Clusters: If you are running a single-node cluster (common for learning), you cannot drain the node without terminating your applications entirely. Ensure you have a multi-node environment for practicing true maintenance workflows.

FAQ

Q: Does draining a node delete my data? A: No, provided your data is stored in a Persistent Volume. If you are using local ephemeral storage, that data will be lost during eviction.

Q: How do I upgrade the cluster version? A: While drain is part of the process, upgrading the control plane and nodes usually involves specific orchestration tools (like kubeadm or managed cloud services). For more advanced automation, consider exploring Kubernetes Cluster API to scale these maintenance tasks.

Recap

Maintenance is a core part of Kubernetes operations. By using cordon to stop new scheduling and drain to safely migrate active workloads, you ensure high availability during your cluster upgrades and server patches. Keeping your infrastructure updated is just as important as keeping your code updated.

Up next: We will begin our journey into Helm, the package manager for Kubernetes, to simplify how we deploy and manage complex applications.