Fractal Techware

Alert runbooks /

KubeQuotaExceeded

A namespace is consuming more of a resource than its ResourceQuota allows, which the admission check should normally make impossible.

   
Severity warning
Source kube-state-metrics v2.x
Key metric kube_resourcequota (labels namespace, resourcequota, resource, type)

What it means

Kubernetes enforces quotas only when objects are created or updated. Existing objects are never removed to satisfy a quota. So usage above the hard limit means the limit changed after the fact, or objects got in without going through quota admission. The alert fires when this state persists.

Consequences: every new pod or object for that resource is rejected until usage falls back below the limit, which can freeze deployments, scale-ups and Jobs for the whole namespace.

Common causes

First checks

  1. Show used and hard values for the namespace and spot the resource where used is higher:
    sum by (resourcequota, resource, type) (kube_resourcequota{namespace="<ns>"})
    
  2. Inspect the quota and when it changed:
    kubectl -n <ns> describe resourcequota <quota>
    kubectl -n <ns> get resourcequota <quota> -o yaml --show-managed-fields | grep -E 'manager:|time:'
    
  3. Check whether new pods are being rejected:
    kubectl -n <ns> get events --field-selector reason=FailedCreate
    
  4. List the largest consumers of the exceeded resource:
    kubectl -n <ns> top pods --sort-by=cpu
    kubectl -n <ns> get pods -o custom-columns='POD:.metadata.name,REQ_CPU:.spec.containers[*].resources.requests.cpu,REQ_MEM:.spec.containers[*].resources.requests.memory'
    
  5. Check your GitOps repository history for the quota change.

Fixing it

Decide which side is wrong. If the new limit is intentional, bring the namespace under it by scaling down or right-sizing requests, then confirm new pods can be created. If the change was accidental, restore the previous spec.hard in its source of truth, so a sync does not revert your fix.