Alert runbooks /

LokiDiscardedSamples

Loki has been dropping incoming log lines for a tenant, and those logs are gone.

What it means

When a distributor or ingester rejects log entries, it counts them by tenant and reason. The client gets a 4xx (often 429 or 400) and usually does not retry, so the entries are lost. The alert fires when discards continue for a sustained period rather than during a single burst.

The reason label is the whole diagnosis. Common values:

• rate_limited: tenant exceeded ingestion_rate_mb / ingestion_burst_size_mb.
• per_stream_rate_limit: one stream exceeded per_stream_rate_limit.
• stream_limit: tenant hit max_global_streams_per_user.
• greater_than_max_sample_age: timestamps older than reject_old_samples_max_age.
• too_far_in_future: timestamps ahead of the clock beyond creation_grace_period.
• line_too_long: line exceeds max_line_size.

Common causes

• A noisy application or a log loop suddenly increasing volume.
• High-cardinality labels (pod IDs, request IDs) creating too many streams.
• An agent replaying a backlog of old files after a restart.
• Wrong clocks on nodes, or timestamps parsed from the log body incorrectly.
• Stack traces or JSON blobs logged as a single huge line.

First checks

1. Break discards down by tenant and reason:

   sum by (tenant, reason) (rate(loki_discarded_samples_total[5m]))
   

2. Check the effective limits for the tenant:

   kubectl -n <loki-namespace> port-forward svc/<loki-or-distributor> 3100
   curl -s localhost:3100/config | grep -E 'ingestion_rate_mb|ingestion_burst_size_mb|per_stream_rate_limit|max_line_size|reject_old_samples|max_global_streams'
   curl -s localhost:3100/runtime_config
   

3. For rate or stream limits, find the heaviest streams with LogCLI or Grafana:

   logcli series '{namespace="<namespace>"}' --analyze-labels
   

4. Read the distributor logs; rejected pushes name the stream and reason:

   kubectl -n <loki-namespace> logs <distributor-pod> --since=15m | grep -iE 'discard|rate limit|too long|too old'
   

Fixing it

Remove high-cardinality labels in the agent config. Raise limits per tenant in the runtime overrides when growth is legitimate. Fix clocks or timestamp parsing for age-related reasons. For long lines, truncate in the agent or raise max_line_size (with max_line_size_truncate to keep the start of the line instead of dropping it).

Related alerts

• LokiRequestErrors: server-side push failures.
• LokiRequestLatency: slow ingestion that may precede client backlogs.