Fractal Techware

Guides /

Redact PII in the OpenTelemetry Collector with OTTL

Tested with otel/opentelemetry-collector-contrib:0.161.0: otelcol validate plus a smoke test that sent real spans and logs and checked the debug output.

Auto-instrumentation records more than you expect: user.email on spans, raw request headers, and log lines like payment by jane@example.com card=4111.... Once that reaches your tracing or logging backend it is copied, indexed and kept for months. The collector is the last place you control before that happens, so it is a good place for a safety net.

This guide uses the transform processor and OTTL to:

The config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

  transform/redact_pii:
    error_mode: ignore
    trace_statements:
      - context: span
        statements:
          # 1. Credentials: remove the attribute entirely.
          - delete_matching_keys(span.attributes, "(?i)(^|\\.)(authorization|cookie|set-cookie|x-api-key|api[_-]?key|password|passwd|secret|access[_-]?token)$$")
          # 2. Emails anywhere in string attribute values and in the span name.
          - replace_all_patterns(span.attributes, "value", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")
          - replace_pattern(span.name, "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")
          # 3. Card-like numbers: 13-19 digits, optionally separated by spaces or dashes.
          - replace_all_patterns(span.attributes, "value", "\\b\\d(?:[ -]?\\d){12,18}\\b", "<card>")
    log_statements:
      - context: log
        statements:
          - delete_matching_keys(log.attributes, "(?i)(^|\\.)(authorization|cookie|set-cookie|x-api-key|api[_-]?key|password|passwd|secret|access[_-]?token)$$")
          - replace_all_patterns(log.attributes, "value", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")
          - replace_all_patterns(log.attributes, "value", "\\b\\d(?:[ -]?\\d){12,18}\\b", "<card>")
          # String bodies: emails, cards and bearer tokens inside free text.
          - replace_pattern(log.body, "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>") where IsString(log.body)
          - replace_pattern(log.body, "\\b\\d(?:[ -]?\\d){12,18}\\b", "<card>") where IsString(log.body)
          - replace_pattern(log.body, "(?i)bearer\\s+[A-Za-z0-9._~+/=-]+", "Bearer <token>") where IsString(log.body)

  batch:
    timeout: 5s

exporters:
  otlp_grpc:
    endpoint: backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/redact_pii, batch]
      exporters: [otlp_grpc]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform/redact_pii, batch]
      exporters: [otlp_grpc]

The important parts

Delete credentials, don’t mask them. delete_matching_keys removes the attribute entirely. A masked Authorization header is still noise in storage. The key regex is anchored to the whole name or its last dot-separated segment ((^|\.)...$). Without that anchor, a loose pattern such as token also deletes useful attributes like gen_ai.usage.input_tokens. Our smoke test includes that attribute to make sure it survives.

$$ instead of $. The collector expands ${...} environment variables in the config file, and $$ is the escape for a literal $. Writing $$ in YAML gives OTTL the regex end anchor $.

Double backslashes. The statements are YAML strings that contain OTTL strings. \\. in the file becomes \. in the regex.

replace_all_patterns(..., "value", ...) rewrites every string attribute value that matches. Integer and double attributes are not touched (see pitfalls).

where IsString(log.body) skips structured (map) bodies. replace_pattern on a map body would do nothing, and the condition makes the intent explicit.

context: groups. The statements are grouped by OTTL context (span, log). This is the current, unambiguous form. If you later add span event statements, give them their own context: spanevent group. Mixing both in one flat list makes the whole list run in the span-event context, and then it only touches spans that have events.

Order. The processor runs before batch and before any exporter. No unredacted copy leaves the process, including through a second exporter on the same pipeline.

Verify it

Use the debug exporter so you can see the output. Save as debug.yaml:

exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      exporters: [debug]
    logs:
      exporters: [debug]

docker run --rm -p 4318:4318 -v "$PWD:/cfg" otel/opentelemetry-collector-contrib:0.161.0 \
  --config=/cfg/config.yaml --config=/cfg/debug.yaml

Send a log record:

curl -s -H 'Content-Type: application/json' http://localhost:4318/v1/logs -d '{
 "resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"checkout"}}]},
 "scopeLogs":[{"logRecords":[{"severityNumber":9,
  "body":{"stringValue":"payment by jane.doe@example.com card=4111111111111111 header Authorization: Bearer eyJhbGciOi.abc-123"},
  "attributes":[{"key":"db.password","value":{"stringValue":"hunter2"}},
                {"key":"customer","value":{"stringValue":"bob@shop.io"}}]}]}]}]}'

The collector prints:

Body: Str(payment by <email> card=<card> header Authorization: Bearer <token>)
Attributes:
     -> customer: Str(<email>)

db.password is gone. For spans, we sent http.request.header.authorization, http.request.header.cookie, user.email, payment.card: "4111 1111 1111 1111", a free-text note with a dashed card number and an email, and a span named lookup jane.doe@example.com. The output kept http.route, order.id: "12345678" and gen_ai.usage.input_tokens: 42 unchanged. The credential attributes were removed and everything else became <email> or <card>.

Make this a regression test: keep a few sample payloads in your repo, run the collector in CI, and grep the debug output for anything that looks like an email.

Pitfalls

Next steps