Kubernetes : Kubernetes Node Management, Maintenance, Delete

Kubernetes Node Management is a crucial part of a Kubernetes cluster. In terms of node maintenance, we all need to accept that the compute resources such as physical servers or cloud instances need to have a downtime for maintenance works such as hardware changes, Operating system upgrade or any other maintenance work. Same theory applies to the Kubernetes Cluster Nodes and lets see how we can perform the Kubernetes Node Management, Maintenance and editing PODs without interrupting to it’s services.

THE KUBECTL DRAIN COMMAND

According to the Kubernetes documentation the drain command can be used to “safely evict all of your pods from a node before you perform maintenance on the node,” and “safe evictions allow the pod’s containers to gracefully terminate and will respect the PodDisruptionBudgets you have specified”.

kubectl drain does following two things:

1. Cordons the node

This part is quite simple, cordoning a node means that it will be marked unschedulable, so new pods can no longer be scheduled to the node. If we know in advance that a node will be taken from the cluster (because of maintenance, like a kernel update, or because we know that there will be scaling in the node), cordoning is a good first step. We don’t want new pods scheduled on this node and then taken away after a few seconds. For example, if we know two minutes in advance that a spot instance on AWS will be terminated, new pods shouldn’t be scheduled on that node, then we can work towards gracefully scheduling all the other pods, as well. On the API level, cordoning means patching the node with node.Spec.Unschedulable=true.

2. Evicts or deletes the pods

After the node is made unschedulable, the drain command will try to evict the pods that are already running on that node. If eviction is supported on the cluster (from Kubernetes version 1.7) the drain command will use the Eviction API that takes disruption budgets into account, if it’s not supported it will simply delete the pods on the node.

Commands for this task:

kubectl drain <n> --delete-local-data --force --ignore-daemonsets # Stop all pods on <n>
kubectl delete node <name> # Remove <node> from the cluster

Interesting Articles on Kubernetes:

Kubernetes : Kubernetes Node Management, Maintenance, Delete
How to add a New Worker Node to a existing kubernetes Cluster
MinIO Client Installation and Quickstart
PLEG is not healthy: Kubernetes Worker Node is in "NotReady" state
Backup MySQL databases in Kubernetes
How to Run Automated Tasks in Kubernetes with a cronjob
How to Completely remove Kubernetes

Remove an unresponsive node from your Kubernetes cluster.

Get a list of your cluster nodes.

kubectl get nodes

Following is a sample output:

NAME STATUS ROLES AGE VERSION
172.16.151.126 Ready etcd,management,master,proxy 39d v1.11.1+icp-ee
172.16.151.182 NotReady worker 39d v1.11.1+icp-ee <<<<<<< unresponsive node
172.16.155.135 Ready worker 39d v1.11.1+icp-ee

Delete the unresponsive node from your cluster.

kubectl delete node <node-IP-address>

Following is a sample command and output:

kubectl delete node 172.16.151.182
node "172.16.151.182" deleted
Remove the IP address of the unresponsive node from the <installation_directory>/cluster/hosts file.

Kubernetes Node Management :

Lets see how we can perform the Cluster Maintenance work without interrupting the running Services

Evict PODs and Safely Perform A Node Maintenance
View the Kubernetes node status :

kubectl get nodes

To check the running POD status and the respective nodes :

kubectl get pods -o wide

Kubernetes Node Management : Node and POD Status , Disable schedule

Executing below the command you can safely take a node out from the cluster and running PODs will be evicted from the node

kubectl drain [NODE_HOSTNAME] --ignore-daemonsets

Kubernetes Node Management : “uncordon” the node

Once, the operation completed we can “uncordon” the node and schooling will be enabled again

kubectl uncordon <NODE_NAME>

Deleting A Node From The Cluster

If we are planning to delete a node from the Kubernetes cluster, we have to drain, evict the running PODs and disable the POD scheduling first.

Kubernetes Node Management : Delete a node

kubectl delete node [NODE_NAME]

How to add a New Worker Node to a existing kubernetes Cluster

Kubernetes : Kubernetes Node Management, Maintenance, Delete