BlackboxProbeFlapping

A blackbox probe keeps alternating between passing and failing instead of settling in one state.


Severity	warning
Source	blackbox_exporter `/probe`
Key metric	`probe_success`

What it means

The alert counts how often probe_success changes value over a recent window. It fires when the probe toggles several times, which usually means some requests fail and others succeed.

Flapping rarely reaches the threshold of a hard outage alert, yet users experience it as random errors or slow page loads. It is often the earliest visible symptom of an overloaded backend or a bad node behind a load balancer.

Common causes

One unhealthy backend behind a load balancer, so probes fail whenever they land on it.
Probe timeout set close to the normal response time, so small latency spikes become failures.
Intermittent DNS failures or multiple A records where one address is dead.
Resource saturation on the service (connection pool exhausted, CPU throttling, GC pauses).
Rate limiting or WAF rules that block the exporter’s traffic part of the time.

First checks

Visualise the pattern over the last few hours:
```
probe_success{instance="<target>"}
```
Regular intervals hint at cron jobs or health-check cycles; random ones hint at load.

Compare probe duration with the module timeout:

max_over_time(probe_duration_seconds{instance="<target>"}[15m])

Check whether DNS returns several addresses and whether one is failing:

dig +short <hostname>
for ip in $(dig +short <hostname>); do curl -s -o /dev/null -w "$ip %{http_code} %{time_total}\n" --resolve <hostname>:443:$ip https://<hostname>/; done

Run the debug probe several times and compare failures:

curl -s "http://<blackbox-exporter>:9115/probe?target=<target>&module=<module>&debug=true" | grep -iE "error|fail|status"

Check backend error rates and pod restarts for the service behind the target.

Fixing it

Remove or repair the unhealthy backend, fix readiness probes so the load balancer stops sending traffic to it, and address saturation. If probes fail only because they time out just above normal latency, give the module a realistic timeout and keep a separate latency alert.

BlackboxProbeFailed: the same probe, failing continuously.
BlackboxSlowProbe: slowness that often precedes timeouts and flapping.
BlackboxDnsLookupSlow: unreliable DNS is a common flapping source.