Fractal Techware

Alert runbooks /

KubeHpaMaxedOut

A HorizontalPodAutoscaler has been pinned at its maximum replica count, so it has no room left to absorb more load.

   
Severity warning
Source kube-state-metrics v2.x (autoscaling/v2)
Key metrics kube_horizontalpodautoscaler_status_current_replicas, kube_horizontalpodautoscaler_spec_max_replicas

What it means

The workload is running exactly maxReplicas pods and has stayed there for a sustained period. HPAs with min equal to max are ignored, since those are fixed on purpose. The HPA’s ScalingLimited condition will usually show TooManyReplicas.

It matters because the autoscaler is no longer protecting you: if load keeps rising, latency and error rates will climb and nothing will react.

Common causes

First checks

  1. Rank HPAs by how close they are to their ceiling:
    sort_desc(kube_horizontalpodautoscaler_status_current_replicas / on (namespace, horizontalpodautoscaler) kube_horizontalpodautoscaler_spec_max_replicas)
    
  2. See the metric values against targets and the scaling conditions:
    kubectl -n <ns> get hpa <hpa>
    kubectl -n <ns> describe hpa <hpa>
    
  3. Check actual usage against requests:
    kubectl -n <ns> top pods -l <selector>
    kubectl -n <ns> get deploy <name> -o jsonpath='{.spec.template.spec.containers[*].resources}'
    
  4. Check whether a recent deploy lines up with the change:
    kubectl -n <ns> rollout history deploy/<name>
    
  5. Look at request rate and latency for the service to confirm the load is real (from your ingress or application metrics).

Fixing it

If the load is real and the cluster has capacity, raise maxReplicas. Confirm nodes, quota and downstream dependencies can take the extra pods first. If a release regressed performance, roll back. If requests are unrealistically small, correct them, since utilization targets are percentages of requests. For a wrong custom metric, fix the metrics pipeline before adding replicas.