AlertmanagerClusterDown

Half or more of the Alertmanager replicas in a cluster are failing their Prometheus scrapes.


Severity	critical
Source	Prometheus scrape of Alertmanager
Key metric	`up` for the Alertmanager job

What it means

Prometheus scrapes each Alertmanager replica. When at least half of a cluster’s instances have been down for most of the recent window, the alert fires. A single replica outage in a three-node cluster does not trigger it; losing a majority does.

With most replicas gone, notification delivery depends on the survivors. If all are down, alerts are queued in Prometheus and nobody is paged. Note that this alert may itself not be delivered, which is why an external dead man’s switch matters.

Common causes

Pods evicted or stuck Pending (node pressure, insufficient resources, PVC for the data volume not bound).
A bad config or image upgrade that prevents startup on every replica.
All replicas scheduled on the same node or zone that just failed (no anti-affinity).
Scrape-side problems only: changed port name, ServiceMonitor selector mismatch, NetworkPolicy blocking Prometheus.
OOM kills during a large alert storm.

First checks

Confirm which instances are down:
```
up{job=~".*alertmanager.*"} == 0
```

Check pod state and recent events:

kubectl -n monitoring get pods -l app.kubernetes.io/name=alertmanager -o wide
kubectl -n monitoring describe pod <alertmanager-pod> | tail -n 30

Look at logs from the previous container if it restarted:

kubectl -n monitoring logs <alertmanager-pod> -c alertmanager --previous

If pods look healthy, test the scrape path directly:
```
kubectl -n monitoring port-forward <alertmanager-pod> 9093 &
curl -s localhost:9093/-/healthy
```
Then check the target’s error in Prometheus under Status > Targets.

Fixing it

Restore capacity first: free node resources, fix the PVC, or roll back the config or image that broke startup. If only scraping is broken, fix the ServiceMonitor or NetworkPolicy. Afterwards, add pod anti-affinity or topology spread constraints and a PodDisruptionBudget.

AlertmanagerClusterCrashlooping: replicas restarting rather than staying down.
AlertmanagerMembersInconsistent: survivors will report missing peers.
PrometheusNotConnectedToAlertmanagers: Prometheus has no Alertmanager left to send to.