Alert runbooks /

KubeletTooManyPods

A node is almost at its pod limit, so the scheduler will soon stop placing pods there no matter how much CPU and memory is free.

What it means

Every node advertises a pod capacity, set by the kubelet’s maxPods (110 by default, often lower on managed clusters where it is tied to available pod IPs). The alert fires when the running pods on a node stay very close to that capacity.

Once the limit is hit, new pods fail to schedule there with Too many pods, DaemonSet pods for new agents cannot land, and a rollout that needs surge capacity can stall.

Common causes

• Low maxPods from the CNI: on EKS with the VPC CNI the limit depends on the instance’s network interfaces and IPs, so small instances hold few pods.
• Many small pods (sidecars split into pods, per-tenant workers) packed onto large nodes.
• Uneven scheduling: no topology spread or anti-affinity, so one node attracts most replicas.
• DaemonSet sprawl: many agents using a fixed share of every node’s slots.

First checks

1. Find the fullest nodes and compare with their capacity:

   topk(10, count by (node) (kube_pod_info{node!=""}))
   

   kube_node_status_capacity{resource="pods"}
   

2. Confirm on the node itself:

   kubectl get node <node> -o jsonpath='{.status.capacity.pods}{"\n"}'
   kubectl get pods -A --field-selector spec.nodeName=<node>,status.phase=Running --no-headers | wc -l
   

3. See which namespaces dominate that node:

   kubectl get pods -A --field-selector spec.nodeName=<node> --no-headers | awk '{print $1}' | sort | uniq -c | sort -rn | head
   

4. Look for pods failing to schedule because of the limit:

   kubectl get events -A --field-selector reason=FailedScheduling | grep -i "too many pods"
   

Fixing it

Short term, add nodes (or let the cluster autoscaler do it) and spread workloads with topologySpreadConstraints. Longer term, raise maxPods in the kubelet configuration where the pod CIDR and CNI allow it; on EKS, enable prefix delegation to get more IPs per node. Alternatively consolidate tiny pods, or use larger instance types when the CNI ties pod count to instance size.

Related alerts

• KubeDaemonSetNotScheduled: a full node cannot take a DaemonSet pod.
• KubeClusterCPURequestsHigh: cluster-wide CPU headroom, the other capacity limit.
• KubeNodeCordoned: cordoned nodes push more pods onto the rest.