Alert runbooks /

KubeAPITerminatedRequests

The API server is actively rejecting or cutting off a large fraction of incoming requests instead of serving them.

What it means

When the API server is overloaded, API Priority and Fairness (APF) and the in-flight limits start terminating requests, typically with HTTP 429. The alert fires when terminations make up a substantial share of all incoming traffic for several minutes.

Clients retry with backoff, so the visible symptom is everything getting slow: kubectl hangs, controllers lag, and low-priority workloads such as operators or CI jobs starve first.

Common causes

• A misbehaving controller or operator in a tight loop of LIST or UPDATE calls.
• A burst of pods or nodes starting at once (mass rollout, cluster autoscaler scale-up).
• Priority levels sized too small for legitimate traffic.
• Too few API server replicas, or --max-requests-inflight set low.
• Slow etcd causing requests to hold concurrency slots longer.

First checks

1. See what is being terminated:

   topk(10, sum by (verb, resource, code, component) (rate(apiserver_request_terminations_total{job="apiserver"}[5m])))
   

2. See which APF priority level and flow schema is rejecting:

   sum by (priority_level, flow_schema, reason) (rate(apiserver_flowcontrol_rejected_requests_total[5m]))
   

3. Check queue pressure per priority level:

   sum by (priority_level) (apiserver_flowcontrol_current_inqueue_requests)
   

4. Inspect the APF configuration:

   kubectl get prioritylevelconfigurations,flowschemas
   

5. Find the noisy client. The debug endpoint lists flows per user:

   kubectl get --raw /debug/api_priority_and_fairness/dump_requests
   

Fixing it

Stop or scale down the client in a loop, then fix its code (use informers, add backoff). If the traffic is legitimate, give it its own FlowSchema and PriorityLevelConfiguration with more shares, or add API server capacity. Do not raise global in-flight limits blindly: it can move the overload onto etcd.

Related alerts

• KubeAPILatencyHigh: the overload usually shows as latency first.
• KubeAPIErrorsHigh: timeouts may also surface as 5xx.
• KubeClientErrors: the client side of the same problem.