Alert runbooks /

CPUThrottlingHigh

A container spends a large share of its CPU scheduling periods throttled because it keeps hitting its CPU limit.

What it means

A CPU limit is enforced by the Linux CFS quota: in each 100ms period a container can use only its allotted CPU time, and once it is used up the container waits for the next period. cAdvisor counts periods and throttled periods. This alert fires when a sizeable fraction of periods have been throttled for a sustained stretch.

It is informational because throttling is not always harmful. But it adds latency in bursts, even when average CPU usage looks low, and it often explains mysterious p99 spikes and slow health checks.

Common causes

• CPU limit set too close to (or below) real peak usage.
• Multi-threaded runtimes (JVM, Go, Node worker pools) that burn the whole quota early in each period.
• Runtimes sizing thread pools from the node’s CPU count rather than the container limit.
• Startup or JIT warm-up bursts, or periodic GC, cron-like work inside the process.
• A sidecar sharing the pod with a tight limit.

First checks

1. Rank containers by throttled ratio:

   topk(10,
     sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
     / sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))
   )
   

2. Compare actual usage with the limit:

   sum by (pod, container) (rate(container_cpu_usage_seconds_total{namespace="<ns>", container!=""}[5m]))
   

   kubectl -n <ns> get pod <pod> -o jsonpath='{range .spec.containers[*]}{.name}: {.resources}{"\n"}{end}'
   

3. Check how much time is lost to throttling:

   rate(container_cpu_cfs_throttled_seconds_total{namespace="<ns>", pod="<pod>"}[5m])
   

4. Correlate with latency or probe failures for the same pod:

   kubectl -n <ns> describe pod <pod> | grep -iE 'unhealthy|probe'
   

Fixing it

Raise the CPU limit, or remove it and rely on requests for scheduling, which is a common choice for latency-sensitive services on clusters with sensible node sizing. Make the runtime limit-aware (for example set GOMAXPROCS to match the limit; modern JVMs detect container limits). If throttling is harmless for a batch job, silence or ignore it for that workload.

Related alerts

• KubeContainerMemoryNearLimit: the memory side of limits that are too tight.
• KubeHpaMaxedOut: throttled pods can push CPU-based autoscaling to its ceiling.
• KubePodNotReady: throttling can make readiness probes time out.