Evidence, notifications, and operations

This stops the daemon and edits the log file (it changes the machine). Try it in a practice environment. You take a copy before editing and put it back at the end.

cp /usr/bin/true /tmp/lab-marker
/tmp/lab-marker
sleep 15
sudo roamswitch events verify; echo "rc=$?"
S=$(sudo find /var/lib/roamswitch/exec_log -name 'seg-*.jsonl' | sort | tail -1)
echo "$S"
sudo systemctl stop roamswitch-server
sudo cp -a "$S" /root/seg.bak
sudo sed -i 's/lab-marker/lab-mxrker/' "$S"
sudo roamswitch events verify; echo "rc=$?"
sudo cp -a /root/seg.bak "$S"
sudo roamswitch events verify; echo "rc=$?"
sudo systemctl start roamswitch-server

✓ Hash chain intact (1 segment). No tampering detected (tamper-evident, not tamper-proof).
  The 6 records after the newest checkpoint are not covered until the next checkpoint.
rc=0
/var/lib/roamswitch/exec_log/seg-0000000000000001.jsonl
✗ Hash chain BROKEN at segment 1 (the records before a checkpoint do not match it (edited, removed or reordered)). The log may have been tampered with or truncated.
rc=2
✓ Hash chain intact (1 segment). No tampering detected (tamper-evident, not tamper-proof).
rc=0

How to read it:

• Changing one letter in the record of /tmp/lab-marker was enough for verify to report a broken chain. The exit code is 2, so a monitoring script can act on it
• Putting the copy back made it intact again. With a copy taken first, you can check the original record again
• "The N records after the newest checkpoint" are records no hash covers yet. verify does not notice a change there
• With the daemon stopped, verify printed no such line. On systemctl stop, the daemon writes a last checkpoint before it exits

This changes the forwarding settings and edits the log (it changes the machine). The daemon sends the newest checkpoint hash to the forwarding destination as EXEC-CHECKPOINT, at most once a minute. Its severity is info, so it is not sent with the default forward_min_severity of low. Here you set it to info and forward to a file.

sudo roamswitch server config set forward_file_path /var/log/roamswitch/events.jsonl
sudo roamswitch server config set forward_min_severity info
sudo roamswitch server config set forward_enabled true
sudo roamswitch server restart
sleep 3
/tmp/lab-marker
sleep 75
sudo grep EXEC-CHECKPOINT /var/log/roamswitch/events.jsonl | tail -1

{"host":"ubuntu","event_id":"1a1cd752e28-6bb8-2","ts":"2026-11-12T01:52:51.240Z","source":"exec_recorder","severity":"info","rule":"EXEC-CHECKPOINT","action":"checkpoint","pid":null,"proc":null,"summary":"segment 1 sha256 c740c1ad22c8d6f01d98aabc6484bae2aeefc41887864e5bbe8bdcfde0996c5e"}

Next, acting as the attacker, write a script that deletes the lab-marker records and recomputes the checkpoints.

cat > /tmp/rewrite.py <<'EOF'
# Practice only: delete the records that mention lab-marker and recompute the checkpoints
import hashlib, json, sys
p = sys.argv[1]
lines = open(p, encoding="utf-8").read().splitlines(keepends=True)
lines = [l for l in lines if "lab-marker" not in l]
out, h, off = [], hashlib.sha256(), 0
for l in lines:
    if l.startswith("{") and '"type":"checkpoint"' in l:
        v = json.loads(l)
        v["bytes"], v["hash"] = off, h.hexdigest()
        l = json.dumps(v, separators=(",", ":"), sort_keys=True) + "\n"
    out.append(l); h.update(l.encode()); off += len(l.encode())
open(p, "w", encoding="utf-8").write("".join(out))
EOF

Count whether the forwarded hash is in the local log, before and after the rewrite.

S=$(sudo find /var/lib/roamswitch/exec_log -name 'seg-*.jsonl' | sort | tail -1)
H=$(sudo grep EXEC-CHECKPOINT /var/log/roamswitch/events.jsonl | tail -1 | sed 's/.*sha256 \([0-9a-f]*\).*/\1/')
sudo systemctl stop roamswitch-server
sudo cp -a "$S" /root/seg.bak
sudo grep -c "\"hash\":\"$H\"" "$S"
sudo python3 /tmp/rewrite.py "$S"
sudo grep -c lab-marker "$S"
sudo roamswitch events verify; echo "rc=$?"
sudo grep -c "\"hash\":\"$H\"" "$S"
sudo cp -a /root/seg.bak "$S"
sudo grep -c "\"hash\":\"$H\"" "$S"
sudo systemctl start roamswitch-server

1
0
✓ Hash chain intact (1 segment). No tampering detected (tamper-evident, not tamper-proof).
rc=0
0
1

In order:

1. Before the rewrite, the forwarded hash matched one checkpoint in the local log
2. The script deleted the lab-marker records (0 left)
3. verify still reported the chain intact, because the checkpoints were recomputed
4. The forwarded hash is gone from the local log (0). The mismatch with the hash kept off the machine shows the rewrite
5. After restoring the copy, it is found again

This creates evidence files (it changes the machine). The practice process gets two dummy environment variables. The value of LAB_API_TOKEN is not a real token. snapshot uses the language setting in server.conf, because the daemon runs the same command; the output below was taken with language set to en. verify and prune, which you run at a terminal, follow your LANG.

cp /usr/bin/sleep /tmp/lab-worker
LAB_API_TOKEN=dummy-not-a-real-token LAB_COLOR=blue setsid /tmp/lab-worker 600 >/dev/null 2>&1 < /dev/null &
sleep 1
P=$(pgrep -x lab-worker); echo "PID=$P"
sudo roamswitch-incident-capture snapshot --pid "$P" --no-packets --reason "lab: practice capture"
B=$(sudo find /var/lib/roamswitch/incident-evidence -mindepth 1 -maxdepth 1 -type d | sort | tail -1)
sudo find "$B" -type f | sort
sudo grep LAB_ "$B/process/environ.txt"
sudo cat "$B/process/cmdline.txt"; echo

PID=28946
evidence bundle saved: /var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956 (captured 6, skipped 0)
Only root can open this folder (it may hold credentials or memory contents). To look inside: sudo ls -l /var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956
  [captured] process/status: captured
  [captured] process/cmdline: captured
  [captured] process/maps: captured
  [captured] process/environ: captured (values that look like secrets are replaced with [REDACTED])
  [captured] process/fd_targets: recorded 3 file descriptor link targets (contents not captured)
  [captured] process/ancestry: recorded 3 ancestor processes (up to PID 1, or stopped early if a process vanished / was unreadable)
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/manifest.json
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/process/ancestry.json
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/process/cmdline.txt
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/process/environ.txt
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/process/fd_targets.txt
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/process/maps.txt
/var/lib/roamswitch/incident-evidence/20261112T015427Z-a159370d-28956/process/status.txt
LAB_COLOR=blue
LAB_API_TOKEN=[REDACTED]
/tmp/lab-worker 600

How to read it:

• The bundle goes into a folder named after the capture time. manifest.json records the SHA-256 of each file
• The folder is drwx------ (only root can open it). A normal user's ls fails with Permission denied. Use sudo to look inside
• Only environment variables whose names contain KEY, TOKEN, SECRET or PASSWORD had their values replaced with [REDACTED]. LAB_COLOR is unchanged
• For open files, only the link targets are recorded, not the contents
• Without --no-packets, it also captures about 30 seconds of packets. On a VM without tshark, packet_capture was skipped, and the output named the package to install
• With --memory, it also saves the writable memory of the process to memory.bin. It is off by default. Passwords and keys can be in it, and the file was written with mode 0600

This edits evidence files (it changes the machine). Use only the practice bundle, and restore it from a copy at the end.

B=$(sudo find /var/lib/roamswitch/incident-evidence -mindepth 1 -maxdepth 1 -type d | sort | tail -1)
sudo roamswitch-incident-capture verify "$B"; echo "rc=$?"
sudo cp -a "$B" /root/bundle.bak
sudo sh -c "echo 'PPid: 1' >> $B/process/status.txt"
sudo roamswitch-incident-capture verify "$B"; echo "rc=$?"
NEW=$(sudo sha256sum "$B/process/status.txt" | cut -d' ' -f1)
OLD=$(sudo python3 -c "import json,sys; m=json.load(open(sys.argv[1])); print(next(a['sha256'] for a in m['artifacts'] if a['name']=='process/status'))" "$B/manifest.json")
sudo sed -i "s/$OLD/$NEW/" "$B/manifest.json"
sudo roamswitch-incident-capture verify "$B"; echo "rc=$?"
sudo rm -rf "$B"; sudo cp -a /root/bundle.bak "$B"
sudo roamswitch-incident-capture verify "$B"; echo "rc=$?"

integrity OK: the bundle matches manifest.json
anchor: manifest.json matches the value recorded at capture time
rc=0
✗ verification FAILED: this bundle cannot be shown to be as it was captured
anchor: manifest.json matches the value recorded at capture time
integrity issues found:
  [modified] process/status.txt
rc=1
✗ verification FAILED: this bundle cannot be shown to be as it was captured
anchor: manifest.json was changed after capture
the files match manifest.json, but manifest.json itself changed after capture, so the match proves nothing
rc=1
integrity OK: the bundle matches manifest.json
anchor: manifest.json matches the value recorded at capture time
rc=0

The four results, in order:

1. The bundle as captured matched, both the files and the anchor
2. After adding a line to status.txt, the first line said "verification FAILED", followed by [modified]
3. After rewriting the SHA-256 in manifest.json to match, the files agree with the manifest. The anchor still reported that manifest.json was changed after capture, the first line said "verification FAILED", and the exit code is 1. The last line explains why the file match proves nothing. In a monitoring script, go by the exit code
4. After restoring from the copy, everything matched

An extra file that the manifest does not list shows as [untracked], and a deleted one as [missing]. You can check the two outside records like this.

sudo tail -1 /var/lib/roamswitch/evidence_anchor.log
sudo lsattr /var/lib/roamswitch/evidence_anchor.log
sudo journalctl -t roamswitch-evidence --no-pager -o cat | tail -1

2026-11-12T01:54:27.236040175+00:00 20261112T015427Z-a159370d-28956 4f5fbe2985699ad63ad2dccebd14542ea9e12edd97910d3a81ea7efcc5523ad4 2ed1863972bbeaf612ab120769178e5acd9b9cc6225afac396f768576e16a5a8 2f88202a8ee39806120008dbe182ce1fb1a5a6c1b332c4c94c3d6b66844c9174
-----a--------e------- /var/lib/roamswitch/evidence_anchor.log
bundle=20261112T015427Z-a159370d-28956 manifest_sha256=4f5fbe2985699ad63ad2dccebd14542ea9e12edd97910d3a81ea7efcc5523ad4

An anchor line holds the time, the bundle name, the SHA-256 of manifest.json, the hash of the previous line, and the hash of this line. Because each line carries the previous line's hash, editing or deleting a line in the middle breaks the chain. Here it is the hash of the line for a bundle captured earlier on this VM. For the very first bundle, it is all zeros. The a from lsattr is the append-only attribute. If you forward the system log, the same value is also kept at that destination.

This changes a setting (it changes the machine). As an example, you change the exec log retention exec_recorder_retention_days from 14 to 30. The next section sets it back to 14.

sudo roamswitch server tui

Once it is open, press these keys.

1. 6 for the config screen, then /, type retention_days, and Enter to filter. Only the exec_recorder_retention_days row is left
2. Press Enter to edit
3. Clear it with Ctrl-U, type 0, and press Enter (out of range)
4. Press Enter to edit again, Ctrl-U, type 30, Enter, then press s
5. Review the diff and save with y. On the service screen 7, press R and y to restart. Close with q

│Service: active (running), boot: enabled                                      │
│Host not isolated.                                                            │
│Security score 71/100 (B (Fair))  passed 27/38                                │
│Forwarding: submitted 3  dropped 0  retry queue 0 items/0 KB                  │
│  [OK] file: ok 3 / failed 0  last ok 2026-11-12T01:50:31.646Z                │
│Exec recorder: netlink  recorded 260  alerts 1  dropped 0  hash chain OK      │

[root] Rejected: exec_recorder_retention_days: exec_recorder_retention_days:
Must be between 1 and 3650: '0'

│> *exec_recorder_retention_days 30                                            │
│Maximum age of exec log data in days.                                         │
│Allowed: 1 .. 3650                                                            │
│Applies after restarting roamswitch-server (no live reload).                  │
│On disk: 14                                                                   │
[root] [1 unsaved] Change staged (1 pending). Press s to review and save.

 R┌ Review changes before saving ────────────────────────────────────────────┐
┌ │exec_recorder_retention_days: 14 -> 30                                    │─┐
│ │                                                                          │ │
│ │These settings apply only after restarting roamswitch-server (there is no │ │
│ │live reload).                                                             │ │
│ │The current file is backed up (timestamped) and replaced atomically.      │ │
│ │Enter/y: confirm    Esc/n: cancel                                         │ │

After saving, compare the backup with the current settings from another terminal.

sudo ls /etc/roamswitch | grep bak
B=$(sudo ls -t /etc/roamswitch | grep 'server.conf.bak' | head -1)
sudo diff /etc/roamswitch/$B /etc/roamswitch/server.conf

server.conf.bak-20261112-014944
443c443
< exec_recorder_retention_days=14
---
> exec_recorder_retention_days=30

What this shows:

• The out-of-range value (0) was rejected without being staged. The reason gives the allowed range (1 to 3650), which is also shown at the bottom of the screen.
• The changed row gets a *, and "On disk: 14" shows the saved value. The file does not change until you press s
• Before saving, the old and new values are shown. To cancel, press Esc or n. After n the change stays staged, and s shows it again. To drop the staged change, press U (discard all). The row went back to 14, and the bottom line said "Discarded 1 pending change. The files are unchanged." Saving left a backup named server.conf.bak-<date>-<time>
• A restart is needed to apply it. R on the service screen asks for confirmation before restarting. x (stop) is labeled "disables all protection"
• Opened as a normal user, the setting values were hidden as ? and the screen said read-only, because only root can read server.conf
• The score and the passed count on the dashboard leave out checks that do not apply and checks that could not be judged