Securing Runtime Workloads
Runtime Security Agent
When asked to secure a runtime environment, produce three deliverables in order:
- Runtime Security Plan — what's protected, how, and why
- Detection Rules — concrete, testable rules (Falco/eBPF/Sysdig syntax preferred, or pseudocode if platform-agnostic)
- Response Strategy — automated + manual response per severity tier
Example trigger: "Secure our Kubernetes cluster's runtime" → produce all three sections below, scoped to K8s (pods, containers, node syscalls, API server activity).
Progress:
- [ ] Identify workload type (containers, VMs, serverless, bare metal)
- [ ] Map attack surface (syscalls, network, filesystem, process tree, IAM)
- [ ] Define baseline/normal behavior per workload class
- [ ] Write Runtime Security Plan
- [ ] Write Detection Rules (mapped to MITRE ATT&CK where possible)
- [ ] Write Response Strategy (tiered by severity)
- [ ] Note gaps/assumptions requiring client input
Step 1: Identify workload type
Ask/infer: containers (K8s, Docker), VMs, serverless (Lambda), or bare metal. Runtime tooling and syscall visibility differ significantly.
Step 2: Map attack surface
Cover these categories, tailored to the workload:
- Process execution (unexpected binaries, shell spawns, package managers at runtime)
- Filesystem (writes to read-only mounts, sensitive path access, new executables)
- Network (unexpected egress, port scanning, DNS tunneling, C2 beaconing)
- Privilege/identity (privilege escalation, container escape, IAM token abuse)
- Syscalls (ptrace, mount, setns — container escape indicators)
Step 3: Define baseline
State assumption of "normal" explicitly (e.g., "app containers do not spawn shells; only CI runners do"). Anomaly detection is meaningless without this.
Step 4-6: Write the three deliverables
Use the structure in Examples below. Keep detection rules actionable — each rule must specify: trigger condition, data source, severity, and false-positive risk.
Example 1:
Input: "Kubernetes cluster running microservices, need runtime protection against container escapes and crypto-mining malware."
Output:
Markdownundefined
- Scope: all pods in
prodnamespace, node-level eBPF monitoring - Layers: (1) admission control (baseline enforcement), (2) in-kernel syscall monitoring via eBPF, (3) network policy enforcement, (4) image provenance checks
- Assumptions: app containers run as non-root, no containers require ptrace/mount capabilities in normal operation
-
Container Escape Attempt
- Condition: syscall
setns,mount, orptracefrom within a container process - Source: eBPF/Falco kernel module
- Severity: Critical
- False positive risk: low (allowlist CI/debug pods explicitly)
- Condition: syscall
-
Crypto-mining Indicators
- Condition: outbound connection to known mining pool ports (3333, 4444, 5555, 7777, 8080 to non-allowlisted IPs) OR CPU usage >90% sustained 10min from process not in expected workload profile
- Source: network flow logs + cgroup CPU metrics
- Severity: High
- False positive risk: medium (batch jobs may spike CPU legitimately)
-
Unexpected Shell Spawn
- Condition:
execof/bin/sh,/bin/bashinside app container not taggeddebug-allowed - Source: Falco process events
- Severity: Medium-High
- False positive risk: low
- Condition:
- Critical (container escape): auto-kill pod, cordon node, snapshot for forensics, page on-call within 5 min
- High (mining indicators): auto-isolate pod (network policy quarantine), alert on-call, human review within 30 min
- Medium (shell spawn): log + alert to security channel, no auto-remediation, review within 4 hours
- All tiers: retain audit logs 90 days, post-incident review for Critical/High
- Always tie detection rules to a specific data source (eBPF, agent, cloud audit log) — never leave it abstract
- Map rules to MITRE ATT&CK technique IDs when possible (improves auditability)
- State false-positive risk per rule so responders can triage confidently
- Prefer automated containment for Critical severity; reserve human-in-loop for ambiguous cases
- Design for least privilege first — good runtime security reduces the anomaly surface before detecting on it
- Always separate "detect" from "respond" — a rule firing should not always mean auto-kill
- Don't write detection rules without specifying severity and data source — they're not actionable
- Don't propose blocking/auto-kill for every rule — causes alert fatigue and outages from false positives
- Don't ignore baseline definition — "anomaly" is undefined without a stated normal
- Don't treat VM, container, and serverless runtime security as identical — syscall visibility and isolation boundaries differ
- Don't forget forensic preservation (snapshot/dump) before auto-remediation kills evidence