CoreDNSErrorsHigh

A growing share of DNS responses from CoreDNS are SERVFAIL, meaning the server could not produce an answer.


Severity	warning, critical
Source	CoreDNS `prometheus` plugin (port 9153)
Key metric	`coredns_dns_responses_total` (label `rcode`)

What it means

Every response is counted with its return code. NXDOMAIN (name does not exist) is normal; SERVFAIL means CoreDNS tried and failed, usually because an upstream did not answer or a plugin errored. The alert looks at the SERVFAIL ratio across the cluster. The warning fires when a small but steady fraction fails; the critical fires when the fraction is several times higher and lookups are visibly breaking for workloads.

Applications receiving SERVFAIL generally treat it as a hard failure, so outbound calls, database connections by hostname and webhook deliveries fail.

Common causes

Upstream resolvers down or unreachable (check forward health checks).
DNSSEC validation failures for a specific external domain.
A stub zone or forward block for a private domain pointing at a dead server.
The kubernetes plugin unable to reach the API server, so cluster names fail.
Upstream rate limiting from cloud VPC resolvers under high query volume.

First checks

See the response code mix:

sum by (rcode) (rate(coredns_dns_responses_total[5m]))

Narrow to the zone and server returning failures:

sum by (server, zone) (rate(coredns_dns_responses_total{rcode="SERVFAIL"}[5m]))

Check upstream responses and health:

sum by (to, rcode) (rate(coredns_proxy_request_duration_seconds_count[5m]))
sum by (to) (rate(coredns_proxy_healthcheck_failures_total[5m]))

Read the errors plugin output:

kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 | grep -iE "error|SERVFAIL|timeout"

Reproduce and compare with the upstream directly:

kubectl run dnstest --rm -it --restart=Never --image=busybox:1.36 -- nslookup <failing-name>
dig @<upstream-ip> <failing-name>

Fixing it

Replace or fix the failing upstream in the coredns ConfigMap, remove dead stub zones, and restore API server connectivity if cluster names fail. If a single external domain is broken, the problem is on their side; consider caching negative answers briefly to reduce load. Enable the log plugin temporarily if you need per-query detail.

CoreDNSForwardHealthcheckFailures: an unhealthy upstream behind the errors.
CoreDNSForwardLatencyHigh: upstream timeouts become SERVFAIL.
CoreDNSPanics: panics recovered mid-query also return SERVFAIL.