How to Safely Upgrade a Kubernetes Node: A Step-by-Step Guide

Upgrading Kubernetes nodes is a critical task that ensures your cluster remains secure, performant, and compatible with the latest features. However, it can feel like walking a tightrope — especially in production environments where downtime isn’t an option. In this guide, I’ll walk you through the step-by-step process to upgrade a Kubernetes node while minimizing risks and ensuring your workloads remain unaffected.

Why Upgrade Kubernetes Nodes?

Kubernetes updates bring essential improvements in security, bug fixes, and new features. Upgrading nodes ensures your cluster:

• Adheres to compatibility policies (version skew).
• Maintains security compliance.
• Performs optimally under new workloads.

That said, let’s get into the practical steps.

Step 1: SSH Into the Node

The first step is to log into the target node where the upgrade will take place. Use your preferred method:

ssh user@<node-ip>

Step 2: Validate Cluster Health Before Starting

Before upgrading, ensure that the cluster is in a healthy state to avoid exacerbating any existing issues.

kubectl get nodes kubectl get pods -A kubectl get events -A

Look for NotReady nodes, failing Pods, or unusual events before proceeding.

Step 3: Drain the Node

To safely upgrade the node, workloads running on it must be evicted and rescheduled to other nodes. This prevents disruptions during the upgrade.

1. Mark the Node as Unschedulable: Prevent new workloads from being scheduled:

kubectl cordon <node-name>

1. Evict Workloads: Drain all running Pods (except DaemonSets and static Pods):

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Note: Pods using emptyDir volumes will lose their data.

Step 4: Backup Cluster State

Take a backup of your cluster state to prevent data loss in case of failure. This is particularly important for clusters using etcd.

ETCDCTL_API=3 etcdctl snapshot save backup.db --endpoints=<etcd-endpoint> --cacert=<ca-file> --cert=<cert-file> --key=<key-file>

Adjust the command for your environment as needed.

Step 5: Check Current Versions

Before upgrading, note the current versions of kubeadm, kubelet, and kubectl on the node:

kubectl version kubelet --version kubeadm version

This helps ensure a seamless upgrade process by comparing against available updates.

Step 6: Validate the Upgrade Plan

Run the following command to validate the kubeadm upgrade plan and ensure compatibility:

kubeadm upgrade plan

This step provides details about the available versions and any potential conflicts.

Step 7: Upgrade Using kubeadm

Run the following command to upgrade the node’s Kubernetes components:

kubeadm upgrade node

If the version matches the control plane, Kubernetes skips the upgrade process for that node.

Step 8: Update Package Repositories

Ensure your package manager’s repositories are up-to-date so you can fetch the latest Kubernetes packages:

sudo apt update

Step 9: Check Latest Version Numbers

To verify the latest available versions of kubeadm, kubectl, and kubelet, use:

apt show kubeadm -a | grep Version 
apt show kubectl -a | grep Version 
apt show kubelet -a | grep Version

Step 10: Install the Latest Version

Install the updated versions of Kubernetes components:

sudo apt install -y kubelet=<version> kubectl=<version> kubeadm=<version>

Step 11: Verify the Upgrade

After installation, confirm that the components are running the correct versions:

kubectl version kubelet --version kubeadm version

Step 12: Restart Critical Pods (Optional)

After restarting kubelet, ensure critical system Pods like kube-proxy or CNI (network plugins) are running correctly.

kubectl get pods -n kube-system

Look for any Pods in CrashLoopBackOff or Pending states and restart them if necessary:

kubectl delete pod <pod-name> -n kube-system

Step 13: Restart the kubelet Service

Restart the kubelet service to apply the updates:

sudo systemctl restart kubelet

Step 14: Rejoin the Node (if Necessary)

If the node fails to join the cluster automatically, you can rejoin it manually:

1. On the Control Plane: Create a token and retrieve the join command:

kubeadm token create --print-join-command

1. On the Worker Node: Run the provided join command:

kubeadm join <control-plane-ip>:6443 --token <token> --discovery-token-ca-cert-hash <hash>

1. Restart the kubelet service:

sudo systemctl restart kubelet

Step 15: Uncordon the Node

Once the upgrade is complete, allow the node to schedule workloads again:

kubectl uncordon <node-name>

Step 16: Verify Node Health

Ensure the node has successfully rejoined the cluster and is ready to accept workloads:

kubectl get nodes

Check the status of the node — it should show as Ready.

Key Notes to Remember

• Always follow Kubernetes Version Skew Policies to ensure compatibility between the control plane and nodes.
• Draining nodes is critical to maintaining workload availability during upgrades.
• Keep an eye on resource usage after the upgrade to confirm stability.

Upgrading Kubernetes nodes can feel daunting, but by following these steps, you can execute the process confidently and safely. Keeping your cluster up-to-date ensures security, reliability, and the ability to leverage the latest Kubernetes features.

Did this guide help you? Share your thoughts or questions in the comments below!

Subscribe to this Medium for more Kubernetes and DevOps tips delivered straight to your inbox.