Alert runbooks /

LokiCompactorHasNotSuccessfullyRunCompaction

The Loki compactor has gone several hours without completing a successful compaction run.

What it means

The compactor merges the many small index files written by ingesters into one file per table per day, and it is also what applies retention and log deletion requests. It records the timestamp of its last successful run. Despite the boltdb_shipper prefix, the same metrics are used for TSDB indexes. The alert fires when that timestamp has been stale for hours, well beyond the normal compaction interval.

Nothing breaks immediately, but queries slow down as index files pile up, retention stops deleting old data (storage costs grow), and delete requests are not processed.

Common causes

• Compactor not running: no compactor target deployed, pod crash looping, or OOM killed on large tables.
• Object storage permission errors: the compactor needs list, read, write and delete on the bucket.
• Multiple compactors running at once in older setups, or none holding the ring lock.
• Working directory full: the compactor downloads tables to working_directory on local disk.
• Very large tables that take longer than expected to process.

First checks

1. Check how long it has been, in hours:

   (time() - loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds) / 3600
   

2. See whether runs are attempted and failing:

   sum by (status) (increase(loki_boltdb_shipper_compact_tables_operation_total[6h]))
   

3. Check the compactor pod:

   kubectl -n <loki-namespace> get pods -l app.kubernetes.io/component=compactor
   kubectl -n <loki-namespace> describe pod <compactor-pod> | grep -A5 'Last State'
   

   In single-binary or simple scalable mode, the compactor runs inside the backend or loki pods.

4. Read its logs for the failure:

   kubectl -n <loki-namespace> logs <compactor-pod> --since=6h | grep -iE 'compact|error' | tail -50
   

5. Check local disk: kubectl -n <loki-namespace> exec <compactor-pod> -- df -h.

Fixing it

Fix bucket permissions, give the compactor more memory or a larger volume for its working directory, and make sure exactly one compactor instance runs. Once it succeeds, it catches up on pending tables and retention on its own.

Related alerts

• LokiRequestLatency: uncompacted indexes make queries slower.
• LokiRequestErrors: storage issues that break the compactor often break other routes too.