KubeAPILatencyHigh

The slowest requests to the Kubernetes API server are taking far longer than they should.


Severity	warning
Source	kube-apiserver `/metrics`
Key metric	`apiserver_request_duration_seconds_bucket` (labels `verb`, `resource`)

What it means

The alert looks at tail (p99) latency of ordinary read and write verbs, excluding long-running calls such as exec, logs, port-forward and watches. It fires when that tail has stayed in the multi-second range, far above normal, for a sustained window.

Slow API calls make kubectl sluggish, delay controllers reconciling, and can push leader-election renewals past their deadline, causing controller-manager or scheduler restarts.

Common causes

etcd disk latency (slow fsync on shared or burstable volumes).
Large unpaginated LIST calls, e.g. a tool listing all pods or secrets cluster-wide every few seconds.
Slow admission webhooks adding their timeout to every matching request.
API server CPU or memory pressure, or too few replicas for the cluster size.
Very large objects (huge ConfigMaps, CRDs with big status fields).

First checks

Find which verbs and resources are slow:

topk(10, histogram_quantile(0.99, sum by (verb, resource, le) (rate(apiserver_request_duration_seconds_bucket{job="apiserver", verb!~"WATCH|CONNECT"}[5m]))))

Find heavy LIST traffic:

topk(10, sum by (resource, scope) (rate(apiserver_request_total{job="apiserver", verb="LIST"}[5m])))

Check etcd latency from the API server side:

histogram_quantile(0.99, sum by (operation, type, le) (rate(etcd_request_duration_seconds_bucket[5m])))

Check webhook latency:

histogram_quantile(0.99, sum by (name, le) (rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m])))

Identify noisy clients in the audit log (if enabled) by userAgent, or check API server resource usage:
```
kubectl -n kube-system top pods -l component=kube-apiserver
```

Fixing it

Move etcd to faster, dedicated disks. Get the noisy client to use informers or paginated lists, or cap it with a FlowSchema. Shorten webhook timeoutSeconds and scope their rules tightly. Give the API server more CPU or replicas when it is simply undersized.

KubeAPIErrorsHigh: slow requests often turn into timeouts.
KubeAPITerminatedRequests: the API server starts shedding load.
EtcdHighFsyncDurations: the most common root cause.