Fractal Techware

Alert runbooks /

BlackboxDnsLookupSlow

Resolving the hostname of a probed endpoint is taking far longer than DNS normally should.

   
Severity warning
Source blackbox_exporter /probe
Key metric probe_dns_lookup_time_seconds

What it means

Before connecting, the exporter resolves the target hostname and records how long that took. The alert fires when lookups average several hundred milliseconds over a sustained window. Healthy lookups are typically a few milliseconds when cached.

Every client of that hostname pays the same delay, and slow DNS frequently turns into timeouts. Because the exporter uses its host’s resolver, this alert often points at your own DNS infrastructure rather than the target.

Common causes

First checks

  1. See whether one hostname or all probes are affected:
    topk(10, avg_over_time(probe_dns_lookup_time_seconds[10m]))
    
  2. Time a lookup from the exporter’s environment:
    kubectl -n monitoring exec <blackbox-exporter-pod> -- cat /etc/resolv.conf
    dig <hostname> | grep "Query time"
    

    Run dig from a debug pod in the same namespace if the exporter image has no shell tools.

  3. Compare against the authoritative server directly:
    dig +short NS <zone>
    dig @<authoritative-ns> <hostname> | grep "Query time"
    
  4. If you are in Kubernetes, check cluster DNS latency:
    histogram_quantile(0.99, sum by (le) (rate(coredns_dns_request_duration_seconds_bucket[5m])))
    

Fixing it

Fix the resolver that is slow: scale CoreDNS or add NodeLocal DNSCache, replace a failing upstream, or use fully qualified names (trailing dot) in probe targets to skip search domains. Set preferred_ip_protocol: ip4 in the module if IPv6 is not routable.