CoreDNS is the DNS backbone of every Kubernetes cluster, handling service discovery, pod-to-pod communication, and external name resolution for every workload running on your nodes. When it stops working — or works intermittently — cascading failures hit fast: pods can't find each other, init containers hang waiting on database hostnames, and application health checks start timing out in waves. This guide covers every common failure mode, ranked by how often they appear in production clusters, with the exact commands to diagnose and fix each one.

What CoreDNS Does and Why It Breaks

CoreDNS runs as a Deployment in the kube-system namespace, exposed by the kube-dns Service at a stable ClusterIP (typically 10.96.0.10 on kubeadm clusters). Every pod gets this IP injected into /etc/resolv.conf at startup. The plugin chain in the Corefile processes each query sequentially: errors → health → ready → kubernetes → forward → cache → loop → reload → loadbalance. A misconfiguration in any plugin, or a resource constraint on the Deployment, produces symptoms that look identical from the outside but have very different root fixes.

First Steps: Collect Baseline State

Before touching anything, run these four commands and save the output:

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 kubectl describe configmap coredns -n kube-system kubectl top pods -n kube-system -l k8s-app=kube-dns

Then launch a debug pod to test resolution from inside the cluster network:

kubectl run dns-debug --image=busybox:1.36 --restart=Never -it --rm -- sh # inside the pod: nslookup kubernetes.default nslookup google.com cat /etc/resolv.conf

If nslookup kubernetes.default fails but nslookup google.com works, the problem is in the kubernetes plugin. If both fail, CoreDNS is unreachable. If external resolution is slow but internal is fine, focus on the forward plugin upstream settings.

Root Cause 1: Forward Loop Detection (Most Common)

The single most frequent CoreDNS failure on kubeadm and many self-managed clusters is the loop plugin firing. CoreDNS detects that forwarded queries return to itself — typically because the node's /etc/resolv.conf lists 127.0.0.53 (systemd-resolved stub) or 127.0.1.1, and those addresses route back to CoreDNS on the pod network.

The log entry is unmistakable:

[FATAL] plugin/loop: Loop (127.0.0.1:53 -> :53) detected for zone ".", see https://coredns.io/plugins/loop for more information

The pod enters CrashLoopBackOff and DNS for every pod in the cluster stops within seconds.

Fix option A — override the forward target directly in the Corefile:

kubectl edit configmap coredns -n kube-system

Find the forward line and replace the node-local nameserver with an explicit upstream:

forward . 8.8.8.8 8.8.4.4 { max_concurrent 1000 }

Fix option B — resolve the stub listener conflict on the node itself:

# On each affected node: sudo systemctl disable --now systemd-resolved sudo rm /etc/resolv.conf echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Option A is immediate and cluster-side only. Option B is permanent but requires access to every node. After saving the Corefile, CoreDNS reloads automatically via the reload plugin. Confirm recovery:

kubectl rollout status deployment/coredns -n kube-system kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

Root Cause 2: conntrack Table Overflow (Busy Clusters)

This is the hardest failure to diagnose because CoreDNS logs show nothing — the pods are healthy, memory and CPU look fine, but DNS queries from workloads time out randomly. The culprit is the Linux kernel conntrack table being full, causing UDP DNS packets to be silently dropped before they reach CoreDNS.

Check on each node:

sudo conntrack -C # current entry count sudo sysctl net.netfilter.nf_conntrack_max # if the count is near the max, you found it

Short-term fix — raise the limit and reduce UDP timeout:

sudo sysctl -w net.netfilter.nf_conntrack_max=1048576 sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=10 sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout_stream=10

Persist those values in /etc/sysctl.d/99-conntrack.conf. The proper long-term fix is NodeLocal DNSCache (covered in Root Cause 5), which bypasses conntrack entirely for DNS traffic using NOTRACK iptables rules.

💡 To quickly verify whether DNS queries are actually reaching CoreDNS, use the DNS Propagation Checker to confirm external DNS for your cluster's egress domain is resolving correctly, then cross-reference with CoreDNS metrics: kubectl port-forward -n kube-system svc/kube-dns 9153:9153 and curl localhost:9153/metrics | grep coredns_dns_requests_total.

Root Cause 3: Resource Starvation (OOMKilled or CPU Throttling)

Default CoreDNS resource limits on kubeadm clusters are conservative: 170Mi memory and 100m CPU per replica. On clusters with hundreds of services or high query rates, CoreDNS gets OOMKilled or throttled, which shows up as intermittent SERVFAIL responses rather than total DNS failure.

Check usage and events:

kubectl top pods -n kube-system -l k8s-app=kube-dns kubectl describe pod -n kube-system -l k8s-app=kube-dns | grep -A5 "Limits\|OOMKilled\|Reason"

Patch the Deployment to increase limits:

kubectl patch deployment coredns -n kube-system --type=json -p='[ {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/memory","value":"100Mi"}, {"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"512Mi"}, {"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/cpu","value":"200m"}, {"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/cpu","value":"500m"} ]'

Also consider scaling the Deployment. Two replicas is the minimum for production; for clusters with more than 50 nodes or 500 services, three or four replicas with pod anti-affinity is the right posture.

kubectl scale deployment coredns -n kube-system --replicas=3

Root Cause 4: Corefile Misconfiguration

Custom Corefile edits — or cluster upgrades that clobber your customisations — produce a wide range of symptoms. Common mistakes:

  • Missing kubernetes plugin block — internal service discovery breaks completely; every svc.cluster.local lookup returns NXDOMAIN
  • Wrong cluster domain — if your cluster uses a non-default domain (e.g. cluster.internal) but the Corefile still says cluster.local, all internal DNS fails silently
  • Forward to a dead upstream — external lookups time out if the configured upstream is unreachable from the node network
  • Stale proxy plugin reference — the proxy plugin was removed in CoreDNS 1.5; any config still referencing it crashes the pod immediately

View and validate the current Corefile:

kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}'

A minimal working Corefile for a default kubeadm cluster looks like this:

.:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf { max_concurrent 1000 } cache 30 loop reload loadbalance }

Root Cause 5: NodeLocal DNSCache Not Deployed

NodeLocal DNSCache is a DaemonSet that runs a local resolver on a link-local address (169.254.20.10) on every node. It answers cluster.local queries from node-local memory, bypasses conntrack for external queries via NOTRACK iptables rules, and significantly reduces latency and CoreDNS load across the board.

Without it, every DNS query traverses SNAT, conntrack, kube-proxy, and at least one network hop to reach CoreDNS. On large clusters or nodes with many pods, this is the structural root cause of intermittent 5-second timeouts from the kernel UDP conntrack race condition.

Check if it is deployed:

kubectl get daemonset -n kube-system node-local-dns

If missing, deploy it using the official manifest:

curl -O https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml KUBEDNS=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}) sed -i "s/__PILLAR__DNS__SERVER__/$KUBEDNS/g" nodelocaldns.yaml sed -i "s/__PILLAR__LOCAL__DNS__/169.254.20.10/g" nodelocaldns.yaml sed -i "s/__PILLAR__DNS__DOMAIN__/cluster.local/g" nodelocaldns.yaml kubectl apply -f nodelocaldns.yaml

After deployment, confirm every node has a running pod and that new pods receive the right nameserver:

kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide kubectl run check --image=busybox --restart=Never -it --rm -- cat /etc/resolv.conf # Expected output: nameserver 169.254.20.10

Diagnosing Slow External Resolution

If internal DNS resolves quickly but external names are slow, the issue almost always sits in the forward plugin upstream chain. CoreDNS exposes upstream latency via Prometheus metrics:

kubectl port-forward -n kube-system svc/kube-dns 9153:9153 & curl -s localhost:9153/metrics | grep coredns_forward_request_duration

High p99 values (over 500ms) on the upstream point to network-level latency between CoreDNS pods and the configured nameservers. On cloud-hosted clusters, switching to the VPC-internal resolver almost always fixes this — the VPC resolver is reachable at 169.254.169.253 on AWS, 168.63.129.16 on Azure, and 169.254.169.254 on GCP.

# AWS example — use VPC resolver in Corefile forward block: forward . 169.254.169.253 { max_concurrent 1000 }

Use the DNS Lookup tool to verify what your external DNS records resolve to from outside the cluster, which helps quickly isolate whether a lookup failure originates inside the cluster or at your upstream provider.

NXDOMAIN for Services That Clearly Exist

If pods receive NXDOMAIN for a service that kubectl get svc confirms is running, check three things in order:

  1. Namespace qualification — pods in namespace app resolve my-svc as my-svc.app.svc.cluster.local. Cross-namespace requires my-svc.other-namespace or the full FQDN.
  2. ndots setting — the default ndots:5 in pod resolv.conf appends cluster search domains to short names first. If your application constructs a name with five or more dots, append a trailing dot to force an absolute lookup and skip the search domain expansion.
  3. No ready endpoints — the kubernetes plugin only publishes endpoint records for pods that pass their readiness probe. If a service has zero ready endpoints, the A record exists but the backing address is missing.
# Verify what CoreDNS returns for the service: kubectl exec -it dns-debug -- nslookup my-svc.default.svc.cluster.local # Check endpoints: kubectl get endpoints my-svc -n default kubectl get pods -l app=my-svc -n default

Common Misdiagnoses

Blaming kube-proxy — kube-proxy manages iptables or IPVS rules for Services, not DNS. If nslookup kubernetes.default resolves correctly but a TCP connection to that ClusterIP fails, the problem is kube-proxy or a NetworkPolicy, not CoreDNS.

Blaming the CNI plugin — CNI handles pod-to-pod routing. If DNS resolution returns a valid A record but subsequent HTTP connections fail or time out, suspect network policy enforcement or a CNI misconfiguration, not CoreDNS.

Assuming the Corefile survived an upgrade — cluster upgrade controllers on kubeadm and some managed platforms can overwrite ConfigMap customisations. Always run kubectl get configmap coredns -n kube-system -o yaml and diff against your expected Corefile immediately after any control plane upgrade.

Running a single replica — a single CoreDNS pod that gets evicted during a node drain leaves the entire cluster without DNS for 30 or more seconds. Two replicas with PodAntiAffinity is the absolute minimum for any environment receiving production traffic.

Verification After Fixing

Run a systematic check from inside the cluster to confirm end-to-end resolution is working:

kubectl run verify --image=dnsutils --restart=Never -it --rm -- bash # inside: dig kubernetes.default.svc.cluster.local @10.96.0.10 dig google.com @10.96.0.10 nslookup kube-dns.kube-system.svc.cluster.local # All three should return NOERROR with A records exit

Then confirm CoreDNS error counters are flat in Prometheus metrics. The gauge coredns_dns_responses_total{rcode="SERVFAIL"} should be near zero and not climbing. A stable counter means the fix held.

2026 Notes: IPv6, DoT, and DNSSEC

Kubernetes dual-stack clusters (IPv4 + IPv6) require the ip6.arpa zone in the kubernetes plugin block. Verify it is present in your Corefile. A missing ip6.arpa zone causes AAAA lookup failures on dual-stack pods, which triggers application-level fallback delays of one to three seconds on every new outbound connection.

CoreDNS supports encrypted upstreams via the tls:// scheme in the forward plugin. If your security policy requires DNS-over-TLS to upstreams, configure it explicitly:

forward . tls://8.8.8.8 tls://8.8.4.4 { tls_servername dns.google max_concurrent 1000 }

DNSSEC validation inside Kubernetes clusters is rarely appropriate. Cluster-internal names are not DNSSEC-signed, and enabling strict validation on the forward plugin causes resolution failures when upstreams return unsigned responses. If DNSSEC validation of external records is a hard requirement, place a validating resolver (Unbound, BIND) as the upstream and let CoreDNS forward without attempting independent validation.

For the underlying DNS wire-format and TTL semantics that CoreDNS implements, RFC 1035 (IETF Datatracker) remains the authoritative specification and is worth reading when debugging unexpected truncation or caching behaviour.

Preventing Future Failures

  • Set a PodDisruptionBudget to prevent node drains from evicting all CoreDNS replicas simultaneously: kubectl create pdb coredns-pdb --selector=k8s-app=kube-dns --min-available=1 -n kube-system
  • Add PodAntiAffinity so replicas land on separate nodes — one drain event should never take out all DNS
  • Deploy NodeLocal DNSCache before hitting scale problems, not after symptoms appear
  • Alert on coredns_dns_responses_total{rcode="SERVFAIL"} and coredns_forward_request_duration_seconds p99 above 500ms
  • Pin CoreDNS image versions in your cluster bootstrap configuration and validate Corefile changes in a staging cluster before applying to production
  • After any control plane upgrade, diff the live Corefile against your source-controlled version immediately