Fractal Techware

Alert runbooks /

KubeHpaReplicasMismatch

A HorizontalPodAutoscaler has decided on a replica count, but the workload has been stuck at a different number for a while.

   
Severity warning
Source kube-state-metrics v2.x (autoscaling/v2)
Key metrics kube_horizontalpodautoscaler_status_desired_replicas, kube_horizontalpodautoscaler_status_current_replicas

What it means

The HPA computes a desired replica count from its metrics. Normally the current count catches up within a minute or two. This alert fires when desired and current differ, the HPA is somewhere between its min and max, and the current count has not changed at all for a sustained period. In other words: scaling was requested and nothing happened.

The usual impact is under-provisioning during a load spike: latency rises while the autoscaler “thinks” it has already reacted.

Common causes

First checks

  1. Compare desired and current across HPAs:
    kube_horizontalpodautoscaler_status_desired_replicas - kube_horizontalpodautoscaler_status_current_replicas != 0
    
  2. Read the HPA’s conditions and events:
    kubectl -n <ns> describe hpa <hpa>
    
  3. Look for pods that are not running and why:
    kubectl -n <ns> get pods -l <selector> --field-selector=status.phase=Pending
    kubectl -n <ns> describe pod <pending-pod> | sed -n '/Events/,$p'
    
  4. Check the ReplicaSet for quota or admission errors:
    kubectl -n <ns> describe rs <replicaset> | grep -iE 'FailedCreate|quota|forbidden'
    
  5. Check whether something else is writing replicas (look at managedFields managers):
    kubectl -n <ns> get deploy <name> --show-managed-fields -o yaml | grep -E 'manager:|replicas'
    

Fixing it

Unblock the thing new pods are waiting on: add node capacity or fix the cluster autoscaler, raise the quota, or fix the readiness problem. If a GitOps tool owns spec.replicas, remove that field from the manifest (or ignore it in the sync diff) so the HPA is the single owner.