How to Debug DNS Resolution issue in Kubernetes?

Create a file named dnsutils.yaml with the following contents:

apiVersion: v1
kind: Pod
metadata:
name: dnsutils
namespace: default
spec:
containers:
- name: dnsutils
image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
restartPolicy: Always

Then create a pod using this file and verify its status:

[root@kubernetesmaster ~]# kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
pod/dnsutils created

[root@kubernetesmaster ~]# kubectl get pods dnsutils
NAME READY STATUS RESTARTS AGE
dnsutils 1/1 Running 0 53m
[root@kubernetesmaster ~]#

Once that pod is running, you can exec nslookup in that environment. If you see something like the following, DNS is working correctly.

[root@kubernetesmaster ~]# kubectl exec -ti dnsutils -- nslookup kubernetes.default
Server: 10.96.0.10
Address: 10.96.0.10#53

Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1

[root@kubernetesmaster ~]#

Check the local DNS configuration

Take a look inside the resolv.conf file. It should look like following.

[root@kubernetesmaster ~]# kubectl exec dnsutils cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[root@kubernetesmaster ~]#

Errors such as the following indicate a problem with the coredns/kube-dns add-on or associated Services:

kubectl exec -ti dnsutils -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10#53

nslookup: can't resolve 'kubernetes.default'

Check if the DNS pod is running

Use the kubectl get pods command to verify that the DNS pod is running.

For CoreDNS:

[root@kubernetesmaster ~]# kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
coredns-576cbf47c7-jhwgd 1/1 Running 0 96m
coredns-576cbf47c7-t97b4 1/1 Running 0 95m
[root@kubernetesmaster ~]#

For kube-dns:

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
...
kube-dns-v19-ezo1y 3/3 Running 0 1h

...
If you see that no pod is running or that the pod has failed/completed, the DNS add-on may not be deployed by default in your current environment and you will have to deploy it manually.

Check for Errors in the DNS pod

Use kubectl logs command to see logs for the DNS containers.

For CoreDNS: Expected to see these kind of logs for a healthy pod

[root@kubernetesmaster ~]# for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
.:53
2020/02/14 17:48:13 [INFO] CoreDNS-1.2.2
2020/02/14 17:48:13 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2020/02/14 17:48:13 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
.:5

Is DNS service up?

Verify that the DNS service is up by using the kubectl get service command.

[root@kubernetesmaster ~]# kubectl get svc --namespace=kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 53d

You can verify that DNS endpoints are exposed by using the kubectl get endpoints command.

[root@kubernetesmaster ~]# kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns xxx.xxx.xxx.xxx:53,yyy.yyy.yyy.yyy:53 53d
[root@kubernetesmaster ~]#

How to delete dnsutils pod?

[root@kubernetesmaster ~]# kubectl delete -f https://k8s.io/examples/admin/dns/dnsutils.yaml
pod "dnsutils" deleted
[root@kubernetesmaster ~]#

How to Debug DNS Resolution issue in Kubernetes?