KubeNodeUnreachable

The control plane has lost contact with a node and marked it unreachable, so its pods are being evicted or are in limbo.


Severity	warning
Source	kube-state-metrics v2.x
Key metric	`kube_node_spec_taint` (key `node.kubernetes.io/unreachable`)

What it means

When a kubelet stops posting node status, the node lifecycle controller sets the node’s Ready condition to Unknown and adds the node.kubernetes.io/unreachable taint. The alert fires when that taint has stayed on the node for several minutes, so a brief network blip does not page.

After the default toleration (five minutes), pods from Deployments are recreated elsewhere. StatefulSet pods are not: they stay Terminating on the lost node until someone confirms it is gone. Local volumes on the node are unavailable.

Common causes

Host down: kernel panic, hardware failure, hypervisor issue, or the cloud instance was terminated or stopped.
Network partition between the node and the API server (security group, route, firewall, or load balancer change).
kubelet stopped: crashed, killed by the OOM killer, or failing on an expired client certificate.
Node frozen by extreme memory pressure or a hung disk, so the kubelet cannot run.

First checks

Confirm the node state and how long it has been silent:

kubectl get node <node> -o wide
kubectl describe node <node> | sed -n '/Conditions:/,/Addresses:/p'

Check whether the host is still alive from Prometheus’ point of view. If node_exporter answers but the kubelet does not, the host is up and the kubelet or its API path is the problem:
```
up{instance=~"<node-ip>:.*"}
```
Check the instance in your cloud console or hypervisor (state, system log, status checks).

If you can reach the host, inspect the kubelet:

systemctl status kubelet
journalctl -u kubelet --since "30 min ago" | tail -50

List what was running there, especially stateful pods:

kubectl get pods -A --field-selector spec.nodeName=<node> -o wide

Fixing it

If the host is alive, restart the kubelet or fix its network path or certificate; the taint is removed automatically once status updates resume. If the host is dead, reboot or replace it. If it is permanently gone, kubectl delete node <node> so StatefulSet pods and attached volumes can move. Make sure the machine is really off before doing that.

KubeNodeNotReady: broader not-ready condition, including a kubelet that reports but is unhealthy.
KubeNodeReadinessFlapping: the node keeps dropping out and coming back.
KubeStatefulSetReplicasMismatch: stateful pods stuck on the lost node.