Fractal Techware

Alert runbooks /

KubeletClientCertificateExpiration

The certificate a kubelet uses to authenticate to the API server will expire soon.

   
Severity warning, critical
Source kubelet /metrics
Key metric kubelet_certificate_manager_client_ttl_seconds

What it means

Kubelets authenticate with a client certificate, normally rotated automatically well before it expires. This alert means rotation has not happened. The warning fires when only days of validity remain; critical when it is down to its final hours.

When it expires, the kubelet gets 401 responses: the node goes NotReady, pod status stops updating and, after the eviction timeout, its pods are rescheduled elsewhere. If many nodes were bootstrapped at the same time, they all expire together.

Common causes

First checks

  1. List nodes by remaining lifetime, in days: 
    sort(kubelet_certificate_manager_client_ttl_seconds / 86400)
  2. Check the certificate on the node: 
    sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -subject -enddate
  3. Look for pending CSRs from that node: 
    kubectl get csr --sort-by=.metadata.creationTimestamp | grep -i <node>
  4. Check rotation errors in the kubelet log: 
    journalctl -u kubelet --since "24 hours ago" | grep -iE "certificate|csr|rotat"
  5. Confirm rotation is enabled: sudo grep -i rotate /var/lib/kubelet/config.yaml.

Fixing it

Approve pending client CSRs (kubectl certificate approve <csr>) and fix whatever stopped auto-approval. Enable rotateCertificates and restart the kubelet so it requests a new certificate. If the certificate has already expired, re-join the node: on kubeadm clusters generate a fresh bootstrap token with kubeadm token create --print-join-command, or replace the node from its node group.