BlackboxExporterProbeScrapeFailed
Prometheus is not getting a response from blackbox_exporter for a probe job, so there is no probe result to evaluate.
| Severity | warning |
| Source | Prometheus scrape of blackbox_exporter /probe |
| Key metric | up for the blackbox job |
What it means
For probe jobs, each “target” in Prometheus is really a request to the exporter’s /probe endpoint. up reflects whether that HTTP request to the exporter worked, not whether the probed endpoint is healthy (that is probe_success). The alert fires when those scrapes keep failing for several minutes.
While it fires, probe_success for the affected targets goes stale, so probe alerts cannot fire either. You have a monitoring blind spot rather than a confirmed outage.
Common causes
- blackbox_exporter pod down, crashlooping or OOM-killed.
- Scrape timeout: the probe takes longer than
scrape_timeout, so Prometheus gives up before the exporter answers. - Relabeling mistakes, so
__address__no longer points at the exporter. - Unknown module name in the
moduleparameter, which makes the exporter return an error. - Network policy or Service change blocking Prometheus from port 9115.
First checks
- Find the failing scrapes and the error text in Status > Targets, or query:
up{job=~".*blackbox.*"} == 0 - Check the exporter is running and its config loaded:
kubectl -n monitoring get pods -l app.kubernetes.io/name=prometheus-blackbox-exporter curl -s http://<blackbox-exporter>:9115/metrics | grep blackbox_exporter_config_last_reload_successful - Call the probe exactly as Prometheus does and time it:
time curl -s "http://<blackbox-exporter>:9115/probe?target=<target>&module=<module>&debug=true" | tail -n 20An error about the module means it is missing from the exporter config.
- Compare the probe duration with the job’s
scrape_timeout:scrape_duration_seconds{job=~".*blackbox.*"}
Fixing it
Restore the exporter (resources, config, image). Fix relabeling so __address__ is the exporter and __param_target the endpoint. Keep the module timeout below the scrape timeout; the exporter also honours the timeout header Prometheus sends, reduced by --timeout-offset.
Related alerts
- BlackboxProbeFailed: the probe runs but the endpoint fails.
- BlackboxSlowProbe: slow probes that approach the scrape timeout.
- BlackboxSslCertificateWillExpireSoon: expiry data stops updating while scrapes fail.