KubeletDown

Prometheus has not been able to scrape a single kubelet for a while.


Severity	critical
Source	kubelet `/metrics` and cAdvisor (kube-prometheus-stack job `kubelet`)
Key metric	`up`

What it means

This alert does not fire for one bad node. It fires when there is no healthy kubelet target at all over a sustained window. That is rarely every node dying at once; far more often the kubelet scrape itself is broken cluster-wide.

Either way it matters: kubelet and cAdvisor metrics feed container CPU and memory dashboards, volume usage alerts and pod-level alerts, and all of those go quiet.

Common causes

The kubelet Service that the Prometheus Operator maintains (kube-system/<release>-kubelet) was deleted or has no endpoints.
The ServiceMonitor for kubelet was disabled or its job label changed in a chart upgrade.
Prometheus lost RBAC access to nodes/metrics or nodes/proxy.
A NetworkPolicy, firewall or security group blocks port 10250 from the Prometheus pods.
kubelet TLS or authentication changes (webhook authn disabled, new serving certificates not trusted).
Genuinely all nodes down, for example after a bad node image rollout.

First checks

Are the nodes actually healthy?
```
kubectl get nodes
```
If they are all Ready, the kubelets are alive and this is a scrape problem.
Look at the scrape error in Status → Targets for the kubelet job, or query:
```
count by (job) (up{job=~".*kubelet.*"})
```

Check the Service and endpoints:

kubectl -n kube-system get svc,endpoints -l app.kubernetes.io/name=kubelet

Test access with the Prometheus service account:

kubectl auth can-i get nodes/metrics --as=system:serviceaccount:monitoring:<prometheus-sa>

Fetch metrics through the API server proxy to rule out the kubelet itself:
```
kubectl get --raw /api/v1/nodes/<node>/proxy/metrics | head
```

Fixing it

Restore the kubelet Service (restarting the Prometheus Operator recreates it when kubelet service management is enabled), re-enable the ServiceMonitor, fix the ClusterRole, or open port 10250 from the monitoring namespace. If nodes are really down, treat it as a node outage.

KubeAPIDown: check this first; a dead API server breaks service discovery too.
KubeletPlegDurationHigh: a slow but still running kubelet.
KubeNodeNotReady: per-node kubelet failures.