RoamSwitch Icon RoamSwitch by Lafine Systems Design

Crisis response for server operators

A containment fired: the first hour

Experience, in a simulation, the hour after a server isolates itself on a serious threat. Build the order of confirm, record, decide, release, and watch as judgment.

  • Level: Practice
  • About 35 min
  • 2026-11-03

#The situation

At 03:12 in the morning, a Telegram notification arrives

"The eBPF runtime guard froze a suspicious process and isolated the host." You are the operator on call. Only the maintenance SSH (port 22) is left on the server.

When RoamSwitch detects a serious threat, it contains it automatically: it isolates the network and freezes processes. This module lets you experience, in a simulation, what to do in the hour after that, and in what order.

The details of screens and operations are in The first hour after an incident. These lessons are for building the judgment behind the steps listed there. They use Kernel-level intrusion detection and autonomous isolation, Incident timeline, Automatic evidence capture on containment, and Notification history.

#The basic order of the first hour

A containment fires for a reason. Before you hurry to lift it, follow the same order in every environment.

StepWhat you doWhy
1. ConfirmLook at what was detected, when, and in what orderFirst learn the reason for the containment
2. RecordSave the logs and evidence before you release or rebootAfter a release or reboot, the state while it was running can become unknowable
3. DecideDecide whether it was an operation you recognize or a truly suspicious oneResume it if it is a false positive, not if it is real
4. ReleaseRelease the containment after you confirm it is safeReleasing with the cause still there brings the same impact back
5. WatchAfter release, check that the same detection does not come backRight after a release is when a repeat is most likely
The number "one hour" has a meaning

On Server Edition, a high-confidence isolation (canary tampering, ransomware-like mass rewriting, eBPF Critical / Emergency / Alert, and so on) moves to degraded (DEGRADED) mode after 3600 seconds (one hour) by default if there is neither an acknowledgement (ack) nor a confirmation that the cause is gone. In degraded mode, new connections other than the maintenance SSH port and the management routes of allowed IPs stay blocked. It does not open by itself as time passes.

#Lab: get used to the output in calm times

You cannot read a screen you see for the first time in the middle of a crisis. Once, in calm times, get used to the output with read-only commands.

See frozen processes and the timeline
roamswitch notifications     # notification history for the past week
roamswitch frozen list       # processes the guards have frozen
roamswitch timeline          # the unified incident timeline
Output (a calm-time example, excerpt)
❄️ 凍結中のプロセス

  凍結中のプロセスはありません。

🕒 統合インシデントタイムライン

  2026-11-03T03:05:32.839736466+00:00  [eBPF Guard]  対応中
    Read sensitive file untrusted: Logged & Monitored (guard policy: no containment)
    プロセス: gnome-initial-s (pid 5311)
    親プロセス系譜: systemd ← systemd
    MITRE ATT&CK: T1552 (Unsecured Credentials)
    検知→遮断の所要時間: 30ms
    実施した対応: Logged & Monitored (guard policy: no containment)

The output follows the language of your RoamSwitch settings (this example is in Japanese). How to read it:

  • Frozen processes (凍結中のプロセス): in calm times it says "there are none" (ありません). When something is frozen, its PID is listed here
  • Timeline: for each detection it shows the process, the lineage of parent processes, the related ATT&CK technique, and the time from detection to response (検知→遮断の所要時間). This example is a detection under a "no containment" policy, where only a record was kept (Logged & Monitored)
  • Times are in UTC. They are 9 hours off from Japan Standard Time, so be careful when you match them against notification times

#Simulation: the first hour

In the simulation below, you experience the decisions of the hour after the first report. Each choice shows its result and the reason. The times are the elapsed time from the first report (03:12).

Simulation: The hour after the first report

The situation is a scenario for learning the procedure, not a real incident. The server is a Server Edition that runs a web app. You can reach it over SSH. When unsure, go back to the order "confirm → record → decide → release → watch."

  1. 03:12 · elapsed 0 min1 / 6

    A notification arrives

    A notification reaches your phone. The isolation is already in effect. The server does not answer from outside; only SSH from your laptop gets through. What do you do first?

    • Best Correct. A containment has a reason. First look at the facts, in time order.
    • Avoid Releasing before you confirm the cause lets traffic resume and brings the same impact back. You also have no evidence yet.
    • Avoid After a reboot, the state while it was running can become unknowable. Save the logs and evidence before you reboot or release.
    • Acceptable The isolation holds, but in an hour it moves to degraded, and time passes without any confirmation. You waste the time you could use to learn the facts.
  2. 03:18 · elapsed 6 min2 / 6

    What is frozen

    The notification history says a child process of the web server's user ran an executable under /tmp and tried to connect outside, so the guard froze that process. What do you do with the frozen process?

    • Best Correct. The freeze is a state that does not resume until the evidence is recorded. Keep it untouched here.
    • Avoid Resuming sets the suspicious process running again. Resume only for a false positive, once you know it was an operation you recognize.
    • Avoid It is already stopped by the freeze, so there is no reason to hurry. Terminating it removes the clues before you record the evidence. You terminate it after recording.
  3. 03:25 · elapsed 13 min3 / 6

    Record the evidence

    When the eBPF guard contains a process or isolates the whole host, it creates an evidence bundle in the background (about 30 seconds of packet capture, and a snapshot of /proc/<pid> when the PID is known). This is a best-effort step, though. Where roamswitch-incident-capture is not installed, nothing is created and no error is shown. The packet capture takes about 30 seconds. What do you do?

    • Best Correct. Save the evidence and the timeline before you release or reboot.
    • Acceptable It is created in most cases. But where the capture tool is not installed, nothing is created and you are not told. You need to check that it exists.
    • Avoid Evidence can become impossible to take after a release or reboot. Save it before you investigate.
  4. 03:40 · elapsed 28 min4 / 6

    Check whether you recognize it

    The timeline shows that at 03:08 a child process of the web server ran /tmp/.x/update, at 03:09 it tried an outside connection, and at 03:12 it was frozen and isolated. No maintenance was scheduled, and nobody put this file there. Checking the access logs with sudo accessscope scan --auto --since 24h --report report.md finds a POST to an upload path at 03:07. What is your decision?

    • Best Correct. This is the flow of a real attack. Do not resume; record first, then terminate.
    • Avoid You may call something a false positive only when you know it was an operation you recognize. There is no basis here, and you would be allowing the attack.
    • Acceptable This is the safe-side choice. But the entry point stays open, and in an hour it moves to degraded. You need to think about how you use the time.
  5. 03:50 · elapsed 38 min5 / 6

    What to do with the isolation

    The process is stopped and the evidence is saved. But you have not yet closed the entry point (the upload path). If you release the isolation, new connections return, and you could be entered again by the same route. In 22 minutes, the isolation moves to degraded. What do you do with the isolation?

    • Best Correct. Keep the isolation until the cause is removed. The acknowledgement also marks that you took over the response.
    • Avoid Releasing with the route still open brings traffic back and risks another intrusion. Release after you confirm the cause is removed.
    • Acceptable Even after the move to degraded, new connections other than the maintenance SSH and allowed IPs stay blocked, and it does not open by itself. No big problem arises, but it does not show that the operator took over the response.
  6. 04:10 · elapsed 58 min6 / 6

    Release, and watch

    You closed the entry point, moved the suspicious file out, terminated the process, and saved the evidence. You have confirmed the cause is removed. What do you do after you release?

    • Best Correct. "Watching" is part of the first-hour procedure.
    • Acceptable Right after a release is when the same detection is most likely to come back. Start watching right after the release.
    • Avoid The records and evidence are the material for rebuilding the facts later. Keep them, and tidy up the old ones only after the retention period.
Debrief

Your choices corresponded to this order.

StepWhere it appeared in the simulation
Confirm03:12 confirm in time order / 03:18 check the frozen process
Record03:25 save the evidence and the timeline
Decide03:40 check whether you recognize it, and decide not to resume
Release03:50 decide whether to keep the isolation / 04:10 release after confirming the cause is removed
Watch04:10 watch from right after the release

There are three easy places to stumble: releasing too early, rebooting or terminating before taking evidence, and resuming without a basis (assuming a false positive). Each one breaks, by your own hand, the situation the containment was protecting for you.

Rotating keys and passwords, contacting the parties concerned, and planning the recovery are responses outside RoamSwitch, and are out of scope for these lessons.

#What it cannot see

What these lessons did not verify
  • These lessons are a tabletop exercise. The containment itself was not triggered on a real machine. The first report and the situation at each step are scenarios for learning the procedure
  • The product's behavior at each step is based on what The first hour after an incident says (updated 2026-10-21). The output of the read-only commands was checked on a real machine
  • The procedure is for Server Edition. The Mac and Linux desktop editions are centered on operations on screen, and their procedures differ. Read sections 3 and 4 of that page
  • When the isolation came from ARP spoofing, or when you were hit by ransomware, the procedure differs (sections 8 and 9 of that page)
Check yourself

One hour passed after the isolation fired, and you did nothing. What happens? (a high-confidence isolation on Server Edition)

Show the explanation

It does not open by itself as time passes. While degraded, an acknowledgement (roamswitch server ack) means a release. That is why you return the acknowledgement only after you have confirmed the cause.

Features covered

Related techniques (ATT&CK): T1685

What was verified

Verified on 2026-11-03. Target: Linux 1.17.30 (Server Edition). A tabletop exercise based on 'The first hour after an incident' (updated 2026-10-21). The degrade limit (3600 s), the meaning of the acknowledgement (ack), and when the evidence bundle is created were checked in the product code. The output of the read-only commands was checked on Ubuntu 26.04. The containment was not triggered on a real machine

Command output varies with the product version and the environment. If yours differs, check the version first.