Fractal Techware

Alert runbooks /

BlackboxExporterProbeScrapeFailed

Prometheus is not getting a response from blackbox_exporter for a probe job, so there is no probe result to evaluate.

   
Severity warning
Source Prometheus scrape of blackbox_exporter /probe
Key metric up for the blackbox job

What it means

For probe jobs, each “target” in Prometheus is really a request to the exporter’s /probe endpoint. up reflects whether that HTTP request to the exporter worked, not whether the probed endpoint is healthy (that is probe_success). The alert fires when those scrapes keep failing for several minutes.

While it fires, probe_success for the affected targets goes stale, so probe alerts cannot fire either. You have a monitoring blind spot rather than a confirmed outage.

Common causes

First checks

  1. Find the failing scrapes and the error text in Status > Targets, or query:
    up{job=~".*blackbox.*"} == 0
    
  2. Check the exporter is running and its config loaded:
    kubectl -n monitoring get pods -l app.kubernetes.io/name=prometheus-blackbox-exporter
    curl -s http://<blackbox-exporter>:9115/metrics | grep blackbox_exporter_config_last_reload_successful
    
  3. Call the probe exactly as Prometheus does and time it:
    time curl -s "http://<blackbox-exporter>:9115/probe?target=<target>&module=<module>&debug=true" | tail -n 20
    

    An error about the module means it is missing from the exporter config.

  4. Compare the probe duration with the job’s scrape_timeout:
    scrape_duration_seconds{job=~".*blackbox.*"}
    

Fixing it

Restore the exporter (resources, config, image). Fix relabeling so __address__ is the exporter and __param_target the endpoint. Keep the module timeout below the scrape timeout; the exporter also honours the timeout header Prometheus sends, reduced by --timeout-offset.