Alert runbooks /

KubeAPIErrorsHigh

A noticeable share of requests to the Kubernetes API server are failing with server-side (5xx) errors.

What it means

The alert compares 5xx responses to total API traffic across the cluster. The warning tier fires when a small but persistent fraction of requests fail; critical fires when the error ratio is several times higher. Client errors (4xx) are not counted.

Controllers, operators, CI pipelines and kubectl users all see these failures. Deployments stall, leader elections can flap, and retries add load that makes things worse.

Common causes

• etcd is slow, out of quota or has lost a member, so writes return 500 or 504.
• An aggregated API (commonly metrics-server) is down, producing 503 for its group.
• A failing admission webhook with failurePolicy: Fail returning errors or timing out.
• API server overload: priority and fairness rejecting requests with 429 plus timeouts surfacing as 504.
• A single API server replica unhealthy behind the load balancer.

First checks

1. Break the errors down by what is failing:

   topk(10, sum by (code, verb, resource, group) (rate(apiserver_request_total{job="apiserver", code=~"5.."}[5m])))
   

2. Check whether one replica is responsible:

   sum by (instance, code) (rate(apiserver_request_total{job="apiserver", code=~"5.."}[5m]))
   

3. Look for unavailable aggregated APIs and failing webhooks:

   kubectl get apiservices | grep -v True
   kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
   

4. Read the API server logs for the dominant error:

   kubectl -n kube-system logs -l component=kube-apiserver --tail=200 | grep -Ei "error|timeout|etcd"
   

5. Check etcd latency seen by the API server:

   histogram_quantile(0.99, sum by (operation, le) (rate(etcd_request_duration_seconds_bucket[5m])))
   

Fixing it

Fix the backend that the breakdown points to: restore the broken APIService or its pods, repair or temporarily relax a failing webhook, defragment or compact etcd if it is near quota, or take an unhealthy API server replica out of rotation. If a runaway client is flooding the API, throttle or scale it down.

Related alerts

• KubeAPILatencyHigh: latency and errors usually rise together.
• KubeAggregatedAPIDown: a common source of 503s for one API group.
• KubeAPITerminatedRequests: overload shedding requests.
• EtcdHighCommitDurations: slow etcd behind the errors.