PrometheusErrorSendingAlertsToAlertmanager

Prometheus is failing to deliver a noticeable share of its alert batches to one or more Alertmanagers.


Severity	warning
Source	Prometheus’ own `/metrics` (2.x and 3.x)
Key metrics	`prometheus_notifications_errors_total`, `prometheus_notifications_sent_total` (label `alertmanager`)

What it means

Prometheus pushes firing alerts to every discovered Alertmanager on each evaluation. The alert fires when the error rate for a specific Alertmanager stays above a small percentage of sends for a sustained period.

If you run an Alertmanager cluster and only one member is failing, notifications still go out through the others, so impact is limited. If every Alertmanager label shows errors, alerts are effectively not being delivered.

Common causes

One Alertmanager pod is restarting, overloaded or stuck, while Prometheus still has it in discovery.
Network problems between Prometheus and Alertmanager: NetworkPolicy, service mesh mTLS, proxies.
TLS or auth mismatch after rotating certificates or enabling basic auth on Alertmanager.
API version mismatch: Prometheus 3.x and Alertmanager 0.27+ only speak the v2 API, so an old api_version: v1 setting breaks.
Alertmanager rejecting payloads, for example alerts with invalid labels.

First checks

Find which Alertmanager is failing:

sum by (instance, alertmanager) (rate(prometheus_notifications_errors_total[5m]))

Check whether alerts are also being dropped outright:
```
rate(prometheus_notifications_dropped_total[5m])
```

Read the error Prometheus gets back:

kubectl -n monitoring logs <prometheus-pod> -c prometheus | grep -i "error sending alert"

Test the endpoint from the Prometheus pod:

kubectl -n monitoring exec <prometheus-pod> -c prometheus -- wget -qO- http://<alertmanager>:9093/api/v2/status

On the Alertmanager side, look at its logs and alertmanager_alerts_invalid_total.

Fixing it

Restart or reschedule the unhealthy Alertmanager member, fix the network path or TLS settings, and set api_version: v2 if an old config still says v1. If the failing address is a stale pod IP, check that discovery uses the headless service or endpoints rather than a hardcoded address.

PrometheusNotConnectedToAlertmanagers: no Alertmanager discovered at all.
PrometheusNotificationQueueRunningFull: failing sends make the queue back up.
Watchdog: end-to-end proof that alerts actually arrive.