Alert runbooks /

PrometheusRuleFailures

Prometheus is failing to evaluate one or more alerting or recording rules, which means some of your alerts are blind right now.

What it means

Every evaluation interval, Prometheus runs each rule group. When a rule’s query returns an error instead of a result, the failure counter for that group goes up. The alert fires when that counter keeps increasing, i.e. the rule is broken on every run, not just once during a restart.

It is critical because a failing alerting rule never fires. The outage you are not being paged for may already be happening.

Common causes

• many-to-many matching: a binary operation (/, and, * on(...) group_left) where one side suddenly has duplicate label sets. Typical after an exporter upgrade adds a label, or when two exporters scrape the same target.
• Recording rule collisions: two recording rules (or a rule and a scraped series) write the same series with identical labels, giving “vector contains metrics with the same labelset after applying rule labels”.
• Query limits: evaluations hitting --query.timeout or --query.max-samples on high-cardinality data.
• Remote/Thanos Ruler setups: the query backend is unreachable, so every evaluation errors.

First checks

1. Find the failing rules and their error message. In the UI it is Status → Rule health (Prometheus 2.x: Status → Rules). From the API:

   curl -s http://<prometheus>:9090/api/v1/rules \
     | jq -r '.data.groups[].rules[] | select(.health != "ok") | "\(.name): \(.lastError)"'
   

2. See which group and instance are failing, and since when:

   sum by (instance, rule_group) (increase(prometheus_rule_evaluation_failures_total[15m])) > 0
   

3. Look at the logs for the exact evaluation error:

   kubectl -n monitoring logs <prometheus-pod> -c prometheus | grep -i "evaluating rule"
   

4. Paste the rule’s expression into the expression browser and run it. For “many-to-many” or “duplicate series” errors, drop the operator and query each side with count by (<matching labels>) (...) > 1 to find the duplicated label set.
5. Validate rule files before redeploying: promtool check rules <file>.yml. For behaviour, promtool test rules catches matching errors that a syntax check misses.

Fixing it

Aggregate away the extra label (sum by (...), max by (...)) or constrain the match with on(...)/ignoring(...). For collisions, rename one of the recording rules. If the backend is timing out, simplify the expression or move the heavy part into a recording rule.

Related alerts

• PrometheusMissingRuleEvaluations: rule groups are too slow to finish within their interval.
• PrometheusConfigReloadFailed: a new rule file was rejected, so old rules are still running.
• Watchdog: confirms the alerting pipeline end to end.