EtcdHighNumberOfLeaderChanges
The etcd cluster has changed leaders several times in a short period.
| Severity | warning |
| Source | etcd 3.5+ /metrics (kube-prometheus-stack job kube-etcd) |
| Key metric | etcd_server_leader_changes_seen_total |
What it means
A leader election happens when followers stop receiving heartbeats from the leader within the election timeout. One election after a restart or upgrade is normal. The alert fires when a member has seen repeated leader changes within a short window, which points to an ongoing stability problem.
During each election, writes stall. On Kubernetes that shows up as slow or failed API requests, leader election churn in controllers, and occasional etcdserver: leader changed errors.
Common causes
- Slow disk I/O on the leader, delaying heartbeats (the most common cause).
- Network latency or packet loss between members, especially across zones.
- CPU contention: etcd sharing a node with busy workloads or the API server under load.
- Rolling restarts of control plane nodes, which are expected and usually short-lived.
- Timeouts too tight for the environment (
--heartbeat-interval,--election-timeout).
First checks
- Confirm the churn and whether it is ongoing:
sum by (instance) (increase(etcd_server_leader_changes_seen_total{job=~".*etcd.*"}[1h])) - Check disk latency on every member:
histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))) - Check peer round trip times:
histogram_quantile(0.99, sum by (instance, To, le) (rate(etcd_network_peer_round_trip_time_seconds_bucket[5m]))) - Check CPU on control plane nodes and etcd logs around the elections:
kubectl top nodes -l node-role.kubernetes.io/control-plane kubectl -n kube-system logs etcd-<node> --since=1h | grep -iE 'elected leader|became leader|lost leader|heartbeat'Log lines about sending heartbeats taking too long point at disk or CPU on the leader.
- Rule out planned work: recent node reboots or control plane upgrades.
Fixing it
Put etcd on fast dedicated SSDs and avoid noisy neighbours on the same disk. Reserve CPU for etcd and keep heavy workloads off control plane nodes. For high-latency networks, raise --heartbeat-interval and --election-timeout together (keeping the election timeout around ten times the heartbeat) on all members.
Related alerts
- EtcdHighFsyncDurations: the usual root cause.
- EtcdNoLeader: the failure case when an election does not complete.
- EtcdMembersDown: a restarting member triggers elections.