EtcdDatabaseQuotaLowSpace

The etcd database file is approaching its configured size quota.


Severity	warning, critical
Source	etcd 3.5+ `/metrics` (kube-prometheus-stack job `kube-etcd`)
Key metrics	`etcd_mvcc_db_total_size_in_bytes`, `etcd_server_quota_backend_bytes`, `etcd_mvcc_db_total_size_in_use_in_bytes`

What it means

etcd enforces a backend quota (--quota-backend-bytes, 2 GiB by default). When the database file reaches it, etcd raises a NOSPACE alarm and rejects every write until the space is reclaimed and the alarm is cleared. For Kubernetes, that means the API server can no longer create or update anything.

The warning fires when the database is well into its quota; critical means it is very close and writes could stop at any moment.

The file size only shrinks after defragmentation. Compaction frees space inside the file, but the file stays the same size.

Common causes

Fragmentation: lots of deleted or updated data that was compacted but never defragmented.
Compaction not running: the API server normally compacts every few minutes; a broken or disabled compaction lets history pile up.
Object explosion: huge numbers of Events, Secrets, ConfigMaps or custom resources.
Large objects, such as big ConfigMaps or Helm release Secrets with long history.

First checks

Check usage per member, and how much is reclaimable:

etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes
etcd_mvcc_db_total_size_in_bytes - etcd_mvcc_db_total_size_in_use_in_bytes

Check status and alarms:

# kubeadm layout; adjust pod name, endpoint and cert paths for your setup
e() { kubectl -n kube-system exec etcd-<node> -- etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key "$@"; }
e endpoint status --cluster -w table
e alarm list

Find what fills the database:

topk(15, apiserver_storage_objects)

kubectl get secrets -A --field-selector type=helm.sh/release.v1 --no-headers | wc -l

Fixing it

If the in-use size is much smaller than the total, defragment one member at a time, followers first, by pointing the helper at each member’s pod (etcd-<node>) in turn:
```
e defrag
```

If in-use is also high, compact old revisions first, then defragment:

rev=$(e endpoint status -w json | grep -o '"revision":[0-9]*' | head -1 | cut -d: -f2)
e compact "$rev"

Delete what is actually taking space: old Events, stale custom resources, Helm history (--history-max).
Once below quota, clear the alarm so writes resume: e alarm disarm.

Raising --quota-backend-bytes buys time (etcd recommends staying at or below 8 GiB) but does not fix growth.

EtcdHighCommitDurations: large databases commit more slowly.
EtcdHighNumberOfFailedGRPCRequests: writes rejected under NOSPACE show up as failed requests.