Fractal Techware

Alert runbooks /

KubeletClientCertificateRenewalErrors

A kubelet is trying to rotate its API client certificate and the attempts keep failing.

   
Severity warning
Source kubelet /metrics
Key metric kubelet_certificate_manager_client_expiration_renew_errors

What it means

The kubelet counts every failed attempt to renew its client certificate. The alert fires when that counter keeps rising over a sustained period, so it is a persistent failure, not a single retry.

The current certificate is still valid, which is why this is a warning. It is your head start: if renewals keep failing, the certificate eventually expires and the node drops out of the cluster.

Common causes

First checks

  1. Which nodes are failing:
    sum by (node, instance) (increase(kubelet_certificate_manager_client_expiration_renew_errors[1h])) > 0
    
  2. Read the actual error:
    journalctl -u kubelet --since "2 hours ago" | grep -iE "certificate_manager|csr|rotat"
    
  3. Check the CSRs for that node and their condition:
    kubectl get csr | grep -E "<node>|Pending|Denied"
    
  4. Verify the approval bindings exist:
    kubectl get clusterrolebinding -o wide | grep -i certificatesigningrequests
    
  5. Check time sync on the node: timedatectl status.

Fixing it

Approve the pending CSR to buy time, then fix the root cause: restore the controller manager or its signing flags, recreate the auto-approval ClusterRoleBinding for system:nodes, fix NTP, or restore connectivity to the API server. Watch the error counter stop increasing and the certificate TTL jump back up.