#The situation
"Ransomware canary tripped (host isolation imminent)" arrives, followed by "Ransomware Canary Tripped". You are on call for a Linux server that keeps its business data under /srv.
This playbook covers the decisions and steps from the first alert to recovery. How the detections work and how to tune them is in Ransomware. Deciding what to stop automatically is in Setting your automatic isolation policy. Here you trigger the incident on a disposable VM and practise what to read and in what order to act.
Features used: Mass-encryption detection, Detection with decoy files, Notification history, Incident timeline, Automatic evidence capture on containment.
The order:
| Step | What you do |
|---|---|
| 1. Read | Use the notification history to learn what happened and what was stopped |
| 2. Hold | Acknowledge, so the full isolation holds even past the time cap |
| 3. Investigate | Check the writer, the evidence and the number of damaged files, then end the writer |
| 4. Restore | Verify the backup, restore it, and lift the isolation last |
| 5. Watch | Check that the same detection does not come back |
#Before you start, make sure you can restore
In production, the backup is the only thing you can restore from. In this exercise too, you make a backup and a list of per-file hashes first.
This creates files (a change). The last section removes them. You create two practice files. make-data.sh writes 300 plain-text dummy files. overwrite-demo.py overwrites every file under a folder with random bytes of the same size. It does not encrypt anything and there is no key.
cd ~
cat > make-data.sh <<'S'
#!/bin/sh
# practice: write 300 dummy files with plain-text content
mkdir -p /srv/appdata
for i in $(seq 1 300); do
yes "order $i: dummy record for practice" | head -c 8192 > /srv/appdata/order_$i.csv
done
S
cat > overwrite-demo.py <<'S'
# practice: overwrite every file under a folder with random bytes of the same size
import os, sys, time
for root, dirs, files in os.walk(sys.argv[1]):
for name in sorted(files):
path = os.path.join(root, name)
with open(path, "r+b") as f:
f.write(os.urandom(os.path.getsize(path)))
time.sleep(0.2)
S
sudo sh make-data.sh
sudo tar -czf /var/backups/appdata.tgz -C /srv appdata
cd /srv && sudo sha256sum appdata/* | sudo tee /var/backups/appdata.sha256 > /dev/null; cd ~
ls /srv/appdata | wc -l
sudo journalctl -u roamswitch-server -b | grep -oE 'following these folders.*|Total monitored canaries.*' | tail -2
sudo ls -A /srv300
following these folders roots=["/srv"]
Total monitored canaries: 16
appdata
.roamswitch_security_canary_do_not_delete.pdf
.roamswitch_security_canary_do_not_delete.xlsx
.roamswitch_security_canary_do_not_delete.zip/srv is a folder the bulk-encryption watch follows. It also holds three decoy files. A writer that walks the whole folder will hit a decoy too.
The backup sits outside /srv. That is outside the watch, but on the same disk. In production, keep it on another disk or another server.
#Trigger it and read the alerts
Run the overwrite script as root against /srv. It imitates what happens after someone has gained root on the server.
This isolates the whole host (a change). The SSH maintenance port stays open, but use a disposable environment only. The way back is in the restore section.
sudo setsid -f python3 overwrite-demo.py /srv > /dev/null 2>&1 < /dev/null
sleep 30
sudo roamswitch notifications 2026-11-12T01:38:28.685476048+00:00
Ransomware Canary Tripped
🚨 Ransomware canary file was Tampered / encrypted write
Host: ubuntu (IP: 192.168.64.17)
Decoy: /srv/.roamswitch_security_canary_do_not_delete.pdf
Suspected process: python3 (PID: 1853)
Action: process frozen + host Air-Gap isolated
Timestamp: 2026-11-12 01:38:28
Process: root / /usr/bin/python3.12 / command: python3 overwrite-demo.py /srv (pid 1853)
Parent processes: systemd / Service: tart-guest-agent.service
Look for a free decryption tool for the encrypted files (only if one has been published): No More Ransom https://www.nomoreransom.org/en/decryption-tools.html / Japan National Police Agency https://www.npa.go.jp/bureau/cyber/countermeasures/ransom.html
2026-11-12T01:38:28.600178015+00:00
🚨 Ransomware canary tripped (host isolation imminent)
Decoy file event: Tampered / encrypted write
File: /srv/.roamswitch_security_canary_do_not_delete.pdf
Suspected process: python3 (PID: 1853)
The host is about to be isolated. The outcome will be reported separately after restore (or via the whitelisted path).
Process: root / /usr/bin/python3.12 / command: python3 overwrite-demo.py /srv (pid 1853)
Parent processes: systemd / Service: tart-guest-agent.service
This isolation is not lifted automatically when the process behind it exits. Check it, then release it with: sudo roamswitch emergency-restore. Without an acknowledgement it moves to degraded mode after the hard cap (60 min by default); while degraded, sudo roamswitch server ack releases it.The notification history shows the newest entry first. Read it from the bottom to get the order of events.
- The alert just before the isolation (Detection with decoy files): PID 1853 rewrote the
.pdfdecoy. Its last line says how to release the isolation. It does not open on its own when the writer ends - Canary Tripped: as the "Action" line says, the writer was frozen and the host was isolated. The last line points to published decryption tools
- Process line: both alerts name the same PID and command. Treat them as one event by one writer
The writer walks /srv in name order, so the first file it rewrites is the .pdf decoy. The exec record (sudo roamswitch events search --pid 1853) shows the writer started at 01:38:28.371. The decoy tampering was detected at 01:38:28.379, about 8 milliseconds later, and the writer was frozen at 01:38:28.601. Times are UTC.
By default the bulk-encryption watch (Mass-encryption detection) only notifies (entropy_watch_freeze=false). A tampered decoy, on Server Edition, goes as far as isolating the host. In this exercise the writer was frozen before it reached appdata, so no bulk-encryption alert came. Pointed at /srv/appdata alone, which has no decoy, the same writer raised only the bulk-encryption alert and kept running. Then stop the PID in the alert yourself (see the last decision in this playbook).
#Hold the isolation
While isolated, the SSH maintenance port (22 by default) still lets you in. First, confirm that the isolation is in place and the writer is stopped, then acknowledge.
Acknowledging changes how the isolation is handled (a change). You lift the isolation itself in the restore section.
sudo roamswitch server ebpf | sed -n 5,7p
sudo roamswitch frozen list
P=$(pgrep -f '^python3 overwrite-demo.py /srv'); echo "P=$P"
ps -o pid,stat,cmd -p $P
curl -sS -m 5 -o /dev/null https://example.com; echo "curl rc=$?"
sudo roamswitch server ack Current isolation: [Isolated: host_with_maintenance_ssh]
Isolated PIDs: [1853]
Maintenance ports kept open: [22]
PID 1853 python3
Reason: Canary file tampering
Frozen at: 2026-11-12T01:38:28.601172546+00:00
If it was a false positive: roamswitch frozen resume <PID> (nothing was killed — it resumes where it stopped)
P=1853
PID STAT CMD
1853 Ts python3 overwrite-demo.py /srv
curl: (28) Resolving timed out after 5002 milliseconds
curl rc=28
✓ Acknowledged the current isolation (35s old). While acknowledged it will not auto-degrade or auto-restore; the current isolation stays in place.
To lift it: sudo roamswitch emergency-restoreHow to read it:
host_with_maintenance_ssh: the whole host is isolated. Only port 22 is openTinSTAT: the writer is stopped with SIGSTOP, not ended.frozen resumewould let it continue overwritingcurl rc=28: outbound traffic is blocked- The acknowledgement line says "it will not auto-degrade or auto-restore". From here, the full isolation stays until someone runs
emergency-restore
A tampered decoy is a high-confidence isolation. It does not open on its own, whether the writer has ended or is still frozen. What an acknowledgement changes is what happens after the cap (isolation_hard_cap_secs, 3600 seconds by default).
- The investigation may take more than an hour: acknowledge before the cap. The full isolation then holds past the cap
- The cap passed with no acknowledgement: the isolation moves to degraded. During degraded,
server acklifts the isolation. If you do not want it lifted yet, do not runserver ackwhile degraded - It was your own work (false positive): no acknowledgement is needed. Resume the writer with
frozen resume <PID>and lift the isolation withemergency-restore
On this VM, killing the frozen writer without acknowledging did not lift the isolation.
P=2560
Current isolation: [Isolated: host_with_maintenance_ssh]
Isolated PIDs: [2560 (exited)]
curl: (28) Resolving timed out after 5003 milliseconds
curl rc=28Until the cap, the isolation holds whether or not you acknowledge.
#Investigate the writer and the damage
Keep the isolation and find out what was rewritten, and how much. End the writer only after checking the evidence.
Read-only, plus an export of the timeline (creates /root/incident.jsonl). Only root can open the evidence bundle folder, so the listing runs inside sudo sh -c too.
sudo roamswitch timeline | head -n 32
B=$(sudo sh -c 'ls -d /var/lib/roamswitch/incident-evidence/*/' | tail -1)
sudo roamswitch-incident-capture verify "$B"
sudo cat "$B/process/cmdline.txt"; echo
sudo cat "$B/process/fd_targets.txt"; echo
cd /srv && sudo sha256sum --quiet -c /var/backups/appdata.sha256 2>/dev/null | wc -l; cd ~
sudo roamswitch timeline --export /root/incident.jsonl 2026-11-12T01:39:03.952957272+00:00 [Canary] open
Server: Isolation acknowledged (trigger: ransomware decoy file tampering, phase: full)
action taken: Isolation acknowledged
2026-11-12T01:38:28.379633309+00:00 [Canary] open
Canary tampered: /srv/.roamswitch_security_canary_do_not_delete.pdf (Tampered / encrypted write)
process: python3 (pid 1853)
process ancestry: systemd
MITRE ATT&CK: T1486 (Data Encrypted for Impact)
action taken: Contained (frozen or isolated)
evidence bundle: 20261112T013828Z-6e83371d-1861 manifest sha256:54735c00f8e82fb5 (6 files)
└─ 2026-11-12T01:38:28.871252877+00:00 [Incident Capture] open
incident evidence bundle captured: /var/lib/roamswitch/incident-evidence/20261112T013828Z-6e83371d-1861
process: python3 (pid 1853)
process ancestry: systemd
action taken: evidence saved
evidence bundle: 20261112T013828Z-6e83371d-1861 manifest sha256:54735c00f8e82fb5 (6 files)
2026-11-12T01:37:52.632379632+00:00 [FIM] recorded only
Log tamper baseline was made again
action taken: Logged
integrity OK: the bundle matches manifest.json
anchor: manifest.json matches the value recorded at capture time
python3 overwrite-demo.py /srv
0 -> /dev/null
1 -> /dev/null
2 -> /dev/null
0
Exported 4 records to /root/incident.jsonl as JSON Lines.How to read it:
- Timeline (Incident timeline): the decoy tampering with its containment, the evidence capture, and the acknowledgement line up on one timeline. The acknowledgement is recorded too. The
[FIM]line at the bottom records the log comparison baseline made when the daemon started. It is "recorded only" and unrelated to this incident - Evidence bundle (Automatic evidence capture on containment): at containment, the
/procdetails of the frozen writer (command line, open files, parent processes and more) were saved automatically.verifyconfirms nothing changed since capture - Open files (
fd_targets.txt): at the freeze, only/dev/nullwas open, so no file was left half-written - 0: the number of files that no longer match the backup hashes. The writer was frozen before it reached
appdata, so none of the 300 were rewritten - Process ancestry: here it is
systemd. In production,sshdor the web server's name there is a lead to the way in
When you have finished investigating, end the writer. This ends a process (a change), and an ended process cannot be brought back.
P=$(pgrep -f '^python3 overwrite-demo.py /srv'); echo "P=$P"
sudo kill -9 $P
sleep 420
pgrep -f '^python3 overwrite-demo.py /srv' || echo "no writer"
sudo roamswitch server ebpf | sed -n 5,6p
curl -sS -m 5 -o /dev/null https://example.com; echo "curl rc=$?"P=1853
no writer
Current isolation: [Isolated: host_with_maintenance_ssh]
Isolated PIDs: [1853 (exited)]
curl: (28) Resolving timed out after 5002 milliseconds
curl rc=28The writer is gone, but the isolation stays. The PID now shows "(exited)".
#Restore the files, lift the isolation, and watch
Restore only after the cause is stopped. The order is verify the backup → restore → verify → lift the isolation. If you lift the isolation first, a remaining writer or way in could rewrite the restored files again.
This overwrites files (a change). First extract to another place and check it against the hash list.
sudo rm -rf /var/tmp/restore-test; sudo mkdir /var/tmp/restore-test
sudo tar -xzf /var/backups/appdata.tgz -C /var/tmp/restore-test
cd /var/tmp/restore-test && sudo sha256sum --quiet -c /var/backups/appdata.sha256 && echo "backup: all 300 files match"; cd ~
sudo cp -a /var/tmp/restore-test/appdata/. /srv/appdata/
cd /srv && sudo sha256sum --quiet -c /var/backups/appdata.sha256 && echo "/srv/appdata: all 300 files match"; cd ~backup: all 300 files match
/srv/appdata: all 300 files matchYou confirmed the backup was intact before putting it back into /srv/appdata. After the restore, all 300 files match their original hashes. Nothing was rewritten in this exercise, but in production you cannot know in advance which files were. The restore steps are the same.
Next, lift the isolation. This restores the host's network (a change). The cause could not be confirmed gone, so emergency-restore asks you to type RESTORE. Run without a terminal (from a pipe, for example), it takes no input and exits with 2. The final restart reloads the normal protection rules.
sudo roamswitch emergency-restore
sudo roamswitch server restart
sleep 12
sudo head -c 32 /srv/.roamswitch_security_canary_do_not_delete.pdf; echo
sudo roamswitch notifications | grep -E '^ [^ ]' | sed -n 2,5p
curl -sS -m 5 -o /dev/null https://example.com; echo "curl rc=$?" Current state:
Isolated (trigger: ransomware decoy file tampering, phase: full)
⚠️ It could not be confirmed that the cause of the isolation is gone. If it remains, putting the network back now restarts traffic while it is still affected.
To continue, type RESTORE: RESTORE
🔄 Lifting all emergency isolation and restoring network baseline...
(snip)
These firewall tables were left as they are, as they are not an isolation:
- roamswitch_server: the server's inbound rules (only ssh_ports and allowed_ports are let in; the usual protection)
✓ The network was recovered.
The rules removed here, such as port-scan detection, stay off until a restart. To put them back: sudo systemctl restart roamswitch-server
The cause may still be present (it was not confirmed gone). Check what happened with roamswitch timeline.
🔄 Restarting roamswitch-server service...
✓ roamswitch-server service successfully restarted.
%PDF-1.4 ROAMSWITCH_CANARY_INTEG
2026-11-12T01:46:14.883840530+00:00
[Delayed] 📝 Isolation incident acknowledged
2026-11-12T01:46:14.861399030+00:00
[Delayed] Ransomware Canary Tripped
curl rc=0After typing RESTORE at the prompt, the isolation rules come off and it prints "The network was recovered." The usual inbound rules (roamswitch_server) stay, since they are not an isolation. The decoy holds its original content again. The final curl rc=0 shows that outbound traffic is back.
The Notification history now lists alerts marked [Delayed]. These are alerts that could not reach the external channels during the isolation and were sent again after the lift. During an isolation, alerts may not reach your external channels. Treat the server's own notification history as the record.
Right after lifting, check that the same detection does not come back. In production, keep watching for a while after closing the way in and rotating the keys and passwords of any account that was used.
This removes files (a change). In production, keep /root/incident.jsonl and the evidence bundle.
sleep 60
sudo journalctl -u roamswitch-server --since -75s | grep -c -e burst -e 'TAMPERED'
sudo roamswitch server ebpf | sed -n 5p
sudo rm -rf /srv/appdata /var/tmp/restore-test /var/backups/appdata.tgz /var/backups/appdata.sha256
rm -f ~/make-data.sh ~/overwrite-demo.py0
Current isolation: [Not isolated]In that minute, there were no bulk-encryption or decoy-tampering records, and the host is not isolated.
If only folders without a decoy are rewritten, you get only the bulk-encryption alert. The host is not isolated and the writer is not stopped.
- You do not recognise the PID in the alert: stop it at once with
sudo kill -STOP <PID>. Stopping keeps the evidence without ending the process. Then continue from "Investigate" in this playbook - You recognise it (your own backup, a legitimate job of the application): check that the amount and place of writing are as usual, record it, and close
- You cannot wait for a person every time: set
entropy_watch_freeze=trueand restart. From then on the writer is frozen automatically (Setting your automatic isolation policy)
- Run
server ackafter the move to degraded, meaning to keep the isolation. An acknowledgement during degraded lifts it - Resume the frozen writer with
frozen resume. Unless it was a false positive, the overwrite continues - Lift the isolation before restoring the backup. If the way in is still open, the restored files can be rewritten again
- Restore over the data without verifying the backup. A broken backup also overwrites the files that were still good
#What you cannot see
- What was written before the freeze is lost. The practice writer hit a decoy first, so no file in
appdatawas rewritten (in both the Japanese and the English run). A writer that rewrites business files before any decoy changes those files until it hits one. Only a backup brings them back - This playbook does not trigger the move to degraded after the cap. What was checked: after the writer ended, with or without an acknowledgement, the isolation held for 7 minutes. Degraded is tried in Setting your automatic isolation policy with a 120-second cap
- The evidence bundle has no packet capture here. This VM has no
tshark, and the bundle records it asskipped. Only the process details were saved - There is no way in to trace in this exercise. The writer was started from the VM's management channel. In production, trace it from the parent processes and the login and web logs
- The practice writer only overwrites with random bytes. How writing ciphertext to new files and deleting the originals, or renaming over them, shows up is covered in Ransomware
A tampered decoy isolated the host and the writer is frozen. The investigation will take about 2 hours. How do you keep the full isolation?
Show the explanation
An isolation with no acknowledgement moves to degraded after the cap (60 minutes by default). An acknowledgement during the full isolation keeps it. An acknowledgement during degraded lifts it.
The writer is stopped and the evidence is checked. Which order is right?
Show the explanation
Verify the backup before restoring, and verify the result afterwards. Lifting the isolation comes last.
#What to learn next
- How the detections work, adding watched folders, and recording random bytes: Ransomware
- Whether to freeze and isolate automatically: Setting your automatic isolation policy
- How to run the first hour after an isolation: A containment fired: the first hour
- Detection with decoy files
- Notification history
- Incident timeline
- Automatic evidence capture on containment
Related techniques (ATT&CK): T1486
Verified on 2026-11-12. Target: Linux 1.17.39. On disposable Ubuntu 24.04 VMs with the published 1.17.39, a root writer overwrote /srv; the decoy-tampering alerts, the freeze and isolation, the acknowledgement, the evidence bundle check, killing the writer and waiting 7 minutes (with and without an acknowledgement), restoring from backup, releasing by typing RESTORE at a terminal and checking with curl were run in Japanese and English on separate VMs, and the bulk-encryption alert was checked by pointing the writer at a folder with no decoy.
Command output varies with the product version and the environment. If yours differs, check the version first.