KubeQuotaExceeded
A namespace is consuming more of a resource than its ResourceQuota allows, which the admission check should normally make impossible.
| Severity | warning |
| Source | kube-state-metrics v2.x |
| Key metric | kube_resourcequota (labels namespace, resourcequota, resource, type) |
What it means
Kubernetes enforces quotas only when objects are created or updated. Existing objects are never removed to satisfy a quota. So usage above the hard limit means the limit changed after the fact, or objects got in without going through quota admission. The alert fires when this state persists.
Consequences: every new pod or object for that resource is rejected until usage falls back below the limit, which can freeze deployments, scale-ups and Jobs for the whole namespace.
Common causes
- The quota’s
spec.hardwas reduced (by a platform team, a GitOps sync or a policy controller) below current usage. - A ResourceQuota was created in a namespace that already had workloads.
- A quota with a new
scopesorscopeSelectorsuddenly started counting existing pods. - Objects created while the quota controller or admission plugin was unavailable.
First checks
- Show used and hard values for the namespace and spot the resource where used is higher:
sum by (resourcequota, resource, type) (kube_resourcequota{namespace="<ns>"}) - Inspect the quota and when it changed:
kubectl -n <ns> describe resourcequota <quota> kubectl -n <ns> get resourcequota <quota> -o yaml --show-managed-fields | grep -E 'manager:|time:' - Check whether new pods are being rejected:
kubectl -n <ns> get events --field-selector reason=FailedCreate - List the largest consumers of the exceeded resource:
kubectl -n <ns> top pods --sort-by=cpu kubectl -n <ns> get pods -o custom-columns='POD:.metadata.name,REQ_CPU:.spec.containers[*].resources.requests.cpu,REQ_MEM:.spec.containers[*].resources.requests.memory' - Check your GitOps repository history for the quota change.
Fixing it
Decide which side is wrong. If the new limit is intentional, bring the namespace under it by scaling down or right-sizing requests, then confirm new pods can be created. If the change was accidental, restore the previous spec.hard in its source of truth, so a sync does not revert your fix.
Related alerts
- KubeQuotaFullyUsed: at the limit rather than over it.
- KubeQuotaAlmostFull: approaching the limit.
- KubeHpaReplicasMismatch: autoscalers cannot add pods while the quota is exceeded.