Alert runbooks /

BlackboxSlowProbe

Probes of an endpoint still succeed, but on average they take much longer than a healthy endpoint should.

What it means

probe_duration_seconds is the total time the exporter spent on a probe, including DNS, connecting, TLS and reading the response. The alert fires when the average stays above roughly a second for a sustained period.

Users are probably seeing the same delay. Slow probes also sit close to the module timeout, so this is often the prelude to flapping or outright failures.

Common causes

• Slow application responses: database contention, cold caches, saturated worker pools.
• Slow DNS resolution on the exporter host.
• TLS handshake delays (OCSP fetching on the server, long certificate chains, CPU-starved TLS terminator).
• Network latency: the exporter probes from a distant region or through a congested proxy.
• The exporter pod itself CPU-throttled while running many concurrent probes.

First checks

1. Break the time down by phase for HTTP probes:

   avg by (phase) (avg_over_time(probe_http_duration_seconds{instance="<target>"}[10m]))
   

   A large resolve points to DNS, connect to network, tls to the handshake, processing to the server.

2. See whether it is one target or many:

   topk(10, avg_over_time(probe_duration_seconds[10m]))
   

   Many slow targets at once usually means the exporter or its network is the problem.

3. Time the request from another location:

   curl -s -o /dev/null -w "dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n" https://<target>/
   

4. Check exporter throttling:

   rate(container_cpu_cfs_throttled_periods_total{container="blackbox-exporter"}[5m])
   

Fixing it

Fix the slow phase: application performance for processing, resolver health for resolve, TLS terminator capacity for tls. If the exporter is the bottleneck, give it more CPU or split probes across instances. If the endpoint is legitimately slow (a heavy report page), probe a lightweight health path instead.

Related alerts

• BlackboxDnsLookupSlow: isolates the DNS part of the delay.
• BlackboxProbeFlapping: slow probes that start hitting the timeout.
• BlackboxProbeFailed: when slowness becomes an outage.