Falco Runtime Security Testing: Container Threat Detection, Rules Coverage Gaps, and Bypass Techniques

Falco Architecture and Detection Model

Falco intercepts Linux system calls via a kernel module or eBPF probe, compares them against a rules engine, and generates alerts. Understanding the detection model reveals where gaps exist.

Rules Coverage Gap Analysis

# Enumerate all active Falco rules and their coverage
# Connect to Falco's gRPC API (if enabled)
falco --list  # Lists all available fields

# Get active rules from running Falco
kubectl exec -n falco daemonset/falco -- falco --list -r /etc/falco/falco_rules.yaml 2>&1 | \
  grep "^- rule:" | sort

# Check which MITRE ATT&CK techniques have no coverage
# Falco tags rules with mitre_* tags
kubectl exec -n falco daemonset/falco -- cat /etc/falco/falco_rules.yaml | \
  grep -A2 "tags:" | grep "mitre_" | sort | uniq -c | sort -rn

# Identify rules in WARN mode (generates alert but doesn't assert detection)
cat /etc/falco/falco_rules.yaml | grep -B5 "priority: WARNING" | grep "rule:"

# List rules with exceptions (potential bypass paths)
cat /etc/falco/falco_rules.yaml | \
  python3 -c "
import yaml, sys
rules = yaml.safe_load(sys.stdin)
for r in rules:
    if isinstance(r, dict) and r.get('exceptions'):
        print(f\"Rule: {r['rule']} — {len(r['exceptions'])} exceptions\")
        for exc in r['exceptions']:
            print(f\"  Exception: {exc}\")
"

Common Coverage Gaps in Default Rules

Bypass Techniques

Fileless Execution via memfd_create

# Many Falco rules look for file writes in /tmp, /dev/shm, etc.
# memfd_create creates an anonymous memory file — no filesystem path
# This bypasses rules that check for writes to suspicious paths

# Demonstrate memfd_create execution (authorized testing only)
python3 << 'EOF'
import ctypes, os, mmap

# Create anonymous memory file (no path on filesystem)
memfd = ctypes.CDLL(None).memfd_create("", 0)
print(f"[*] memfd fd: {memfd}")

# Write shellcode/payload to memory fd
with open("/bin/ls", "rb") as f:
    payload = f.read()

os.write(memfd, payload)
print(f"[*] Wrote {len(payload)} bytes to memfd")

# Execute via /proc/self/fd/
os.execve(f"/proc/self/fd/{memfd}", ["ls"], os.environ)
EOF

# Test if Falco detects this (check alerts)
kubectl logs -n falco -l app=falco --since=30s | grep -i "memfd\|proc/self/fd"
# If no alert: detection gap confirmed

Syscall-Level Evasion

# Falco rules match on specific syscalls and their arguments
# Alternative syscalls for the same operation may bypass rules

# Example: File read bypass
# Rule may match: open() / openat() of sensitive files
# Bypass: use openat2() (newer syscall not in all rule sets)
# Or: mmap() a file without open() if already mapped

# Test specific syscall bypass scenarios
# 1. Use ptrace to manipulate another process without triggering ptrace rule
strace -e trace=ptrace -p $(pgrep falco) 2>&1 | head  # observe Falco's own ptrace usage

# 2. Shell spawned from unexpected parent (bypass parent-process rules)
# Falco: "spawning shell in container" checks proc.name in container
# Bypass: rename the shell binary before exec
cp /bin/bash /tmp/notabash
chmod +x /tmp/notabash
/tmp/notabash -c "id"  # Check if Falco detects this vs standard bash

# 3. Network connection via raw socket (bypasses connect() rules)
python3 -c "
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_TCP)
# Raw socket doesn't use connect() — may bypass network-based rules
print('Raw socket created:', s.fileno())
s.close()
"

# 4. File access via /proc/[pid]/mem to read another process's memory
# Bypasses file-based access rules since it's a procfs read
python3 -c "
import os
pid = os.getpid()
with open(f'/proc/{pid}/mem', 'rb') as f:
    pass  # Check if Falco flags this as memory scraping
"

Disabling Falco Detection

# If an attacker has container escape or host privileges, they can silence Falco

# Method 1: Kill the Falco process (creates detection gap)
pkill -9 falco  # Detection gap starts immediately

# Method 2: Unload the kernel module (if not using eBPF)
rmmod falco  # Requires host privileges + no kmod protection

# Method 3: eBPF probe unload
# If eBPF is pinned to BPF filesystem — harder to remove
# But: overwriting the BPF map that stores rules
bpftool map list | grep falco  # Find Falco's BPF maps
bpftool map dump id   # Inspect map contents

# Method 4: Modify rules at runtime via API (if /var/run/falco.sock is writable)
# gRPC API allows rule reload — inject modified rules
falco-client reload --config /etc/falco/falco_rules_minimal.yaml

# Method 5: Flood Falco with events to trigger dropped syscalls
# Falco has a syscall drop threshold — high-rate events cause sampling
# Find threshold:
cat /etc/falco/falco.yaml | grep -i "syscall_event_drops\|buffer_size"

Container Escape Detection Testing

# Test whether Falco detects various container escape techniques
# Use authorized test environment — never test against production

# Test 1: Docker socket access detection
# Falco rule: "Launch Sensitive Mount Container"
kubectl run test-escape --rm -it --image=alpine \
  --overrides='{"spec":{"volumes":[{"name":"docker-sock","hostPath":{"path":"/var/run/docker.sock"}}],"containers":[{"name":"test","image":"alpine","volumeMounts":[{"name":"docker-sock","mountPath":"/var/run/docker.sock"}]}]}}' \
  -- sh
# In container: ls /var/run/docker.sock
# Check Falco for: "Launch Sensitive Mount Container" or "Read sensitive file"

# Test 2: Privileged container + host path mount
kubectl run priv-test --rm -it --image=alpine --privileged -- sh -c "ls /proc/1/root/"
# Expected Falco alert: "Launch Privileged Container"

# Test 3: nsenter escape (if host PID namespace accessible)
# In privileged pod:
nsenter --target 1 --mount --uts --ipc --net --pid -- bash
# Expected Falco alert: "Change thread namespace"

# Test 4: Cgroup notify_on_release escape
# In privileged container:
# mkdir /tmp/cgroup_escape
# mount -t cgroup -o memory memory /tmp/cgroup_escape
# mkdir /tmp/cgroup_escape/x
# echo 1 > /tmp/cgroup_escape/x/notify_on_release
# echo /escape_script > /tmp/cgroup_escape/release_agent
# echo $$ > /tmp/cgroup_escape/x/cgroup.procs

# Verify Falco coverage with Atomic Red Team (CNCF-tailored tests)
# kubectl apply -f https://github.com/falcosecurity/event-generator/...
# Use falco-event-generator for automated detection coverage testing
kubectl run event-gen --rm -it \
  --image=falcosecurity/event-generator:latest \
  -- run syscall --loop
# Check Falco output for each triggered event

Alert Pipeline Validation

# Falco generates alerts but they must reach your SIEM/response system
# Test the full pipeline, not just Falco's detection

# 1. Check Falco's output configuration
cat /etc/falco/falco.yaml | grep -A10 "outputs:"
# Verify: file_output, stdout_output, syslog_output, grpc_output, http_output

# 2. Trigger a known-detectable event and trace through the pipeline
# Clear/save current logs
kubectl exec -n falco daemonset/falco -- sh -c "tail -f /var/log/falco/events.log" &

# Trigger a rule: read /etc/shadow in a container
kubectl run shadow-test --rm -it --image=alpine -- cat /etc/shadow
# Should generate: "Read sensitive file untrusted"

# 3. Verify alert reached downstream (Falcosidekick → SIEM)
# Check Falcosidekick metrics
kubectl port-forward -n falco svc/falcosidekick-ui 2802:2802 &
curl localhost:2802/metrics | grep -i "falcosidekick_inputs_total\|falcosidekick_outputs"

# 4. Introduce artificial delay to test MTTD
# Time from event to alert in SIEM:
START=$(date +%s%N)
kubectl run timing-test --rm --image=alpine -- cat /etc/shadow
# In SIEM: find the alert for "Read sensitive file"
END=$(date +%s%N)
echo "MTTD: $(( (END - START) / 1000000 ))ms"

# 5. Test alert under load (does Falco drop syscalls under pressure?)
# Run high-frequency event generator
for i in $(seq 1 1000); do
  kubectl exec test-pod -- cat /dev/null &
done
# Check Falco metrics for dropped_events counter
kubectl exec -n falco daemonset/falco -- \
  curl -s localhost:8765/metrics | grep "falco_scap_ndrops_buffer"

eBPF Probe Security

# Modern Falco uses eBPF probe instead of kernel module
# eBPF programs are verified by the kernel's verifier, but configuration matters

# Check which driver Falco is using
kubectl exec -n falco daemonset/falco -- falco --version 2>&1 | grep -i "driver\|ebpf"

# Check if eBPF probe is pinned (persistent across restarts)
ls -la /sys/fs/bpf/falco/  # Pinned eBPF programs/maps

# Check BPF ring buffer size (affects how many events can be buffered)
kubectl exec -n falco daemonset/falco -- \
  cat /proc/$(pgrep -f "falco")/status | grep VmRSS

# eBPF map inspection (requires host privileges)
bpftool prog list | grep -i falco
bpftool map list | grep -i falco

# Check if privileged containers can access BPF subsystem
# CAP_BPF allows reading BPF maps — could expose Falco's internal state
kubectl run bpf-test --rm -it --image=ubuntu \
  --overrides='{"spec":{"containers":[{"name":"bpf","image":"ubuntu","securityContext":{"capabilities":{"add":["CAP_BPF"]}}}]}}' \
  -- bash -c "apt-get install -y bpftool && bpftool prog list"

Writing Effective Detection Rules

# Falco rule example: Detect memfd_create (memory-only execution)
# /etc/falco/custom_rules.yaml:

- rule: Detect memfd_create Execution
  desc: Detects processes using memfd_create for fileless execution
  condition: >
    spawned_process and
    proc.name = "memfd_create" or
    (syscall.type = execve and fd.name startswith "/proc/self/fd/")
  output: >
    Possible fileless execution detected
    (user=%user.name proc=%proc.name pid=%proc.pid cmdline=%proc.cmdline
    parent=%proc.pname container=%container.name image=%container.image.repository)
  priority: CRITICAL
  tags: [container, mitre_defense_evasion, T1055]

- rule: Suspicious outbound connection from container
  desc: Detects unexpected network connections to non-standard ports
  condition: >
    outbound and
    container and
    not fd.sport in (80, 443, 8080, 8443) and
    not proc.name in (allowed_network_procs)
  exceptions:
    - name: allowed_network_procs
      fields: [proc.name]
      values:
        - [curl]
        - [wget]
  output: >
    Unexpected outbound connection from container
    (proc=%proc.name dest=%fd.rip:%fd.rport container=%container.name)
  priority: WARNING
  tags: [network, container, T1071]

- rule: Write below root in container
  desc: Detects writes to / in a container (indicator of container escape attempt)
  condition: >
    open_write and
    container and
    fd.name startswith "/" and
    not fd.name startswith "/tmp" and
    not fd.name startswith "/var/run" and
    not fd.name startswith "/proc"
  output: >
    Write to root filesystem path in container (path=%fd.name proc=%proc.name)
  priority: CRITICAL
  tags: [container, T1611]

Falco Hardening

# 1. Use eBPF probe over kernel module (more resilient, same coverage)
# falco.yaml:
driver:
  kind: ebpf

# 2. Enable syscall event drop alerting
falco.yaml:
syscall_event_drops:
  threshold: 0.1
  actions: [log, alert]
  rate: 0.03333
  max_burst: 1

# 3. Protect Falco from being killed by privileged containers
# Deploy Falco as a DaemonSet with PodSecurityPolicy / PSS restricted
# Ensure Falco pod itself runs with:
securityContext:
  privileged: true  # Required for kernel module
  readOnlyRootFilesystem: true

# 4. Falcosidekick authentication
# Enable basic auth or mTLS for Falcosidekick webhook
# falcosidekick config:
auth:
  bearer: "YOUR_STRONG_TOKEN"

# 5. Custom rules for application-specific behaviors
# Add rules for: expected processes, expected network destinations,
# expected file access patterns — reduces false negatives

# 6. Test rules with event-generator before deploying to prod
docker run -it falcosecurity/event-generator run syscall
# Validate each rule fires correctly

# 7. Enable JSON output for SIEM integration
# falco.yaml:
json_output: true
json_include_output_property: true

# 8. Monitor Falco's own metrics
kubectl port-forward -n falco svc/falco-metrics 8765:8765
curl localhost:8765/metrics | grep -E "falco_events_processed|falco_scap_ndrops"