Upgrading a server's OS remotely | Learn Defense with RoamSwitch

#The situation

11 p.m.: a kernel update is waiting on a remote server

apt list --upgradable shows a new kernel, openssh-server and sudo. You can only reach the server over SSH. What do you do if the connection drops mid-upgrade, or if RoamSwitch raises an alert?

An OS upgrade rewrites many critical files at once. To a tamper monitor, that looks a lot like an attack. An operator wants to know three things. What the protection reports during the upgrade, and what it does not. How to keep the upgrade running if the connection drops. How to confirm, after the reboot, that the protection really came back.

This module is a runbook. In a disposable VM, it runs an apt full-upgrade that includes the kernel, over SSH, and checks each step.

Step	What you do	What you use
1. Prepare	Create a pending upgrade and an outside workstation	An older kernel, a network namespace, Falco
2. Check	Confirm the baseline is clean and that you will not be locked out	Critical system file tamper monitoring, SSH brute-force detection and blocking, Limiting SSH to allowed sources
3. Upgrade	Run it so that it survives a dropped connection	`tmux`, `nc`
4. Read	Read the alerts, and the changes no alert shows	Notification history, `dpkg -S`, `dpkg --verify`
5. Restore	Reboot, confirm the protection works, clean up	Kernel-level intrusion detection and autonomous isolation, SUID/SGID change monitoring

Updating RoamSwitch itself, and uninstalling it, are covered in Updates, threat data and uninstalling. This module is about upgrading the OS underneath.

Practice in a disposable environment

This module swaps the kernel and reboots the OS. The VM can also be driven with tart exec, so work can continue while SSH is down. On a real host, confirm a way in other than SSH first, such as a cloud console.

#Prepare the practice host

The VM already has a few updates waiting that came out after it was made. For practice, boot it on an even older kernel (7.0.0-38), and take the new kernel, openssh-server and sudo back to older versions. These are package operations only; RoamSwitch's settings are not touched.

Boot the previous kernel and downgrade

This changes the kernel and the boot settings (a change to the system). The boot setting is put back within this lab. The kernel and packages return to the new versions in the upgrade later.

Install the previous kernel, make it the next boot choice, and reboot.

sudo apt-get install -y linux-image-7.0.0-38-generic linux-modules-7.0.0-38-generic
sudo cp /etc/default/grub /etc/default/grub.lab-orig
sudo sed -i 's/^GRUB_DEFAULT=.*/GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 7.0.0-38-generic"/' /etc/default/grub
sudo update-grub
sudo systemctl reboot

After the boot, downgrade the kernel metapackages, remove the new kernel, and put the boot setting back.

uname -r
sudo apt-get install -y --allow-downgrades linux-generic-hwe-24.04=6.8.0-31.31 linux-image-generic-hwe-24.04=6.8.0-31.31 linux-headers-generic-hwe-24.04=6.8.0-31.31
sudo apt-get purge -y linux-image-7.0.0-42-generic linux-modules-7.0.0-42-generic linux-headers-7.0.0-42-generic linux-hwe-7.0-headers-7.0.0-42 linux-hwe-7.0-tools-7.0.0-42 linux-tools-7.0.0-42-generic linux-main-modules-zfs-7.0.0-42-generic
sudo cp /etc/default/grub.lab-orig /etc/default/grub
sudo update-grub
sudo apt-get install -y --allow-downgrades openssh-server=1:9.6p1-3ubuntu13 openssh-client=1:9.6p1-3ubuntu13 openssh-sftp-server=1:9.6p1-3ubuntu13 sudo=1.9.15p5-3ubuntu5
apt list --upgradable 2>/dev/null | grep -c upgradable

Excerpt

7.0.0-38-generic
50

50 packages now have an update waiting.

Install Falco so that Kernel-level intrusion detection and autonomous isolation has a sensor. The steps are the same as in Suspicious execution and malware.

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc \
  | sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" \
  | sudo tee /etc/apt/sources.list.d/falcosecurity.list
sudo apt-get update
sudo FALCO_FRONTEND=noninteractive FALCO_DRIVER_CHOICE=modern_ebpf apt-get install -y falco
sudo roamswitch status --server | grep -A1 "eBPF Runtime"

Excerpt

  [PASS] eBPF Runtime Security Agent - Active (Falco / Tetragon / RoamSwitch)
         Compliance: CIS - / NIST CSF DE.CM-09

Last, create the outside workstation. A network namespace inside the VM plays your PC at 10.99.0.2, which logs in with a key. The key is a practice key made on the spot.

sudo ip netns add outside
sudo ip link add veth-host type veth peer name veth-out
sudo ip link set veth-out netns outside
sudo ip addr add 10.99.0.1/24 dev veth-host
sudo ip link set veth-host up
sudo ip netns exec outside ip addr add 10.99.0.2/24 dev veth-out
sudo ip netns exec outside ip link set veth-out up
ssh-keygen -q -t ed25519 -N "" -C lab-dummy-key -f /var/tmp/lab_key
cat /var/tmp/lab_key.pub >> ~/.ssh/authorized_keys
K="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=5 -o BatchMode=yes -o IdentitiesOnly=yes -i /var/tmp/lab_key"
R="sudo ip netns exec outside ssh $K admin@10.99.0.1"
$R 'echo ok from $SSH_CLIENT'

Output

ok from 10.99.0.2 45148 22

From here on, $R '…' stands for a command typed over SSH from your PC.

#Check before the upgrade

Two things to check first. Is the tamper monitor's baseline clean? Will you stay able to get in during the upgrade?

Check the baseline, the way in, and automatic updates

Read-only, except the final fim update, which rewrites the baseline (a change to the system).

First, compare the tamper monitor's baseline with the files now.

$R 'sudo roamswitch fim verify'

Output

  🔍 Verifying integrity of critical system binaries and configs... (92 files)
  🚨 File modification or tampering detected!
    [modified] /home/admin/.ssh/authorized_keys
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)
  (3 changes look like package upgrades; the daemon takes them into the baseline)

  If these are legitimate changes (e.g. OS upgrade), run 'sudo roamswitch fim update' to baseline.

authorized_keys is the file you added a key to in the previous lab. When you added it, a tampering alert arrived at once and named the writing process. Once you have confirmed it was your change, take it into the baseline.

$R 'sudo roamswitch fim update; sudo roamswitch fim verify'

Output

  📝 Generating baseline database from current critical files...
  ✓ FIM baseline database successfully updated and saved. (91 entries recorded)
  ✓ SUID / SGID baseline updated. (20 entries recorded)

  🔍 Verifying integrity of critical system binaries and configs... (91 files)
  ✓ No tampering detected. All files verified healthy. (91 files monitored)
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)

Next, the way in over SSH: the brute-force guard, the source allow-list and the port sshd listens on.

$R 'sudo roamswitch bans | sed -n 5,10p'
$R 'sudo roamswitch ssh-allow status | sed -n 5,8p'
$R 'sudo roamswitch server config get ssh_ports; sudo ss -ltnpH | grep sshd'

Output

  State     : detects only (does not ban)
  Reason    : fail2ban already guards SSH, so the ban is left to it (ssh_guard_yield_to_fail2ban=true).
  Log       : journal
  Ports     : 22
  The banning is left to fail2ban. The sources the SSH guard never bans (an open SSH session, a recent public-key login, ssh_guard_allow) are not passed on to fail2ban, so add your own addresses to fail2ban's ignoreip (/etc/fail2ban/jail.local) too. What fail2ban bans: sudo fail2ban-client status sshd

  ssh_allow_only_enabled  : false
  ssh_allow_only_sources  : -
  daemon                  : off
  Sources with an SSH session now: 10.99.0.2
22
LISTEN 0      4096               *:22       *:* users:(("sshd",pid=11610,fd=3),("systemd",pid=1,fd=67))

Banning is left to fail2ban, and your own source is one the SSH guard never bans. So that fail2ban does not lock you out either, it tells you to add yours to ignoreip. The allow-list is not in use. ssh_ports and the port sshd listens on are both 22.

Last, check that no automatic update will overlap, and what will change.

$R 'systemctl list-timers --no-pager apt-daily-upgrade.timer apt-daily.timer roamswitch-update.timer'
$R 'sudo fuser -v /var/lib/dpkg/lock-frontend; echo fuser=$?'
$R 'apt-get -s full-upgrade | grep -E "^Inst (openssh-server|sudo) "; apt-get -s full-upgrade | grep -E "^Inst linux-image-7"; apt-get -s full-upgrade | grep -E "^[0-9]+ upgraded"'

Output

NEXT                         LEFT LAST                        PASSED UNIT                    ACTIVATES
Thu 2026-11-12 02:23:22 UTC 11min Tue 2026-11-03 16:25:55 UTC      - apt-daily-upgrade.timer apt-daily-upgrade.service
Thu 2026-11-12 02:42:57 UTC 30min -                                - roamswitch-update.timer roamswitch-update.service
Thu 2026-11-12 11:03:04 UTC    8h Sat 2026-10-31 11:01:41 UTC      - apt-daily.timer         apt-daily.service

3 timers listed.
Pass --all to see loaded but inactive timers, too.
fuser=1
Inst openssh-server [1:9.6p1-3ubuntu13] (1:9.6p1-3ubuntu13.20 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [arm64]) []
Inst sudo [1.9.15p5-3ubuntu5] (1.9.15p5-3ubuntu5.24.04.4 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [arm64])
Inst linux-image-7.0.0-43-generic (7.0.0-43.43~24.04.1 Ubuntu:24.04/noble-updates [arm64])
47 upgraded, 13 newly installed, 0 to remove and 3 not upgraded.

apt-daily-upgrade.timer: when automatic security updates (unattended-upgrades) run. The next run is at 02:23 UTC, 11 minutes away. In this VM the upgrade took about two minutes, so they did not overlap. If there is not enough time, start after that run
roamswitch-update.timer: RoamSwitch's signed data updates. It installs no OS packages
fuser=1: no process holds the dpkg lock now. A 0 means another update is running
The Inst lines: the kernel, sshd and sudo will be replaced

When `fim verify` shows a change

Decide before you start. An upgrade replaces package files. If a tampered file is replaced by the same upgrade, its content matches the package's record and joins the baseline, and you can no longer tell what had changed.

You confirmed the change is yours: take it in with sudo roamswitch fim update, then upgrade
A change you do not recognise: do not start the upgrade. Check the alert history for the writing process and the SSH login. Do not update the baseline until you are done
Nothing is shown: go ahead with the upgrade

Try the way back from a lockout beforehand, too. If isolation fires and SSH stops, you use emergency-restore from the console.

Try `emergency-restore` from the console

This removes firewall rules (a change to the system). The restart at the end puts the rules back.

Run it from the console (tart exec for this VM), not over SSH. Nothing is isolated now, so run it as is first, then with --force to see how it removes rules.

sudo roamswitch emergency-restore
sudo roamswitch emergency-restore --force
$R 'echo ok from $SSH_CLIENT'
sudo nft list tables
sudo systemctl restart roamswitch-server
sleep 3
sudo nft list tables

Excerpt

  Nothing is isolated, and no isolation rule is left, so nothing was removed. To remove the rules that can stop traffic anyway (port-scan and Web guard blocks included): sudo roamswitch emergency-restore --force
  Current state:
    No isolation record (any leftover rules are removed here)
  🔄 Lifting all emergency isolation and restoring network baseline...
    ✓ Isolation records
    ✓ Firewall tables that can stop traffic
    ✓ DNS settings
    ✓ Connection tracking
  These firewall tables were left as they are, as they are not an isolation:
    - roamswitch_server: the server's inbound rules (only ssh_ports and allowed_ports are let in; the usual protection)
  ✓ The network was recovered.
  The rules removed here, such as port-scan detection, stay off until a restart. To put them back: sudo systemctl restart roamswitch-server
ok from 10.99.0.2 58506 22
table inet roamswitch_server
table inet roamswitch_metadata_watch
table inet roamswitch_egress_block
table inet roamswitch_dnstunnel
table inet roamswitch_dev_guard
table inet roamswitch_server
table inet roamswitch_metadata_watch
table inet roamswitch_egress_block
table inet roamswitch_dev_guard
table inet roamswitch_portscan
table inet roamswitch_dnstunnel

With no isolation record and no rule left, it ends without removing anything (exit code 0). With --force it removes the rules that can stop traffic, without asking. The port-scan rules, roamswitch_portscan, went away; the server's inbound rules, roamswitch_server, are not an isolation and were left in place. The open SSH session was not cut. After lifting, restart roamswitch-server as the output says, to put the protection rules back. When an isolation is in place, running it without --force shows what is isolated and removes it after you answer y. emergency-restore does not lift brute-force guard bans; how to lift those is in SSH brute force, and when you lock yourself out.

#Upgrade so that a dropped connection does not stop it

Run apt directly in an SSH session, and a dropped connection stops it too. Run it inside tmux, and it keeps going.

Upgrade inside `tmux` and drop the connection

This upgrades packages (a change to the system). The kernel and packages are not taken back to the old versions.

Record the start time in a file. It is the marker for finding "what changed during the upgrade" later. Separately, start a watcher on your PC that tries a new connection every 5 seconds.

sudo touch /var/tmp/upgrade-start; date -u +%FT%T | sudo tee /var/tmp/T0
sudo ip netns exec outside setsid sh -c 'while :; do if nc -z -w2 10.99.0.1 22; then r=open; else r=refused; fi; echo "$(date -u +%T) $r"; sleep 5; done' >/var/tmp/probe.log 2>&1 < /dev/null &

Output

2026-11-12T02:14:31

Log in over SSH and start the upgrade inside tmux. --force-confold keeps the current configuration files, so a configuration question cannot stall it. After 15 seconds, kill your SSH client to simulate the drop.

sudo ip netns exec outside ssh -tt $K admin@10.99.0.1 'TERM=xterm tmux new -s up "sudo DEBIAN_FRONTEND=noninteractive apt-get full-upgrade -y -o Dpkg::Options::=--force-confold 2>&1 | tee /var/tmp/upgrade.log"' </dev/null >/var/tmp/ssh-client.out 2>&1 &
C=$!
sleep 15
sudo kill $C; sleep 2; kill -0 $C 2>/dev/null && echo "client alive" || echo "client gone"

Output

client gone

The connection is gone, so log in again.

$R 'tmux ls; tail -n 2 /var/tmp/upgrade.log'

Output

up: 1 windows (created Thu Nov 12 02:14:32 2026)
Get:34 http://ports.ubuntu.com/ubuntu-ports noble-updates/main arm64 linux-main-modules-zfs-7.0.0-43-generic arm64 7.0.0-43.43~24.04.1 [2073 kB]
Get:35 http://ports.ubuntu.com/ubuntu-ports noble-updates/main arm64 linux-modules-7.0.0-43-generic arm64 7.0.0-43.43~24.04.1 [285 MB]

You are back in. The tmux session is still there, and the upgrade is still fetching packages; dropping your SSH did not stop it. Check from the console as well.

systemctl is-active ssh ssh.socket
ps -eo pid,etime,args | grep -E "dpkg|apt-get|tmux" | grep -v grep

Excerpt

active
active
  13844       00:16 tmux new -s up sudo DEBIAN_FRONTEND=noninteractive apt-get full-upgrade -y -o Dpkg::Options::=--force-confold 2>&1 | tee /var/tmp/upgrade.log
  13851       00:16 apt-get full-upgrade -y -o Dpkg::Options::=--force-confold

At this point sshd is still running too. It stops later, when openssh-server is replaced. After the upgrade finishes, read the watcher's record and sshd's log.

grep -v '^Connection' /var/tmp/probe.log | uniq -c -f1
sudo journalctl -u ssh -u ssh.socket --since "$(cat /var/tmp/T0 | tr T ' ')" --no-pager -o short-iso | grep -E "Stopped|Listening|Closed"

Output

     10 02:14:31 open
      7 02:15:22 refused
      7 02:15:57 open
2026-11-12T02:15:21+00:00 ubuntu systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
2026-11-12T02:15:21+00:00 ubuntu systemd[1]: Closed ssh.socket - OpenBSD Secure Shell server socket.
2026-11-12T02:15:55+00:00 ubuntu systemd[1]: Listening on ssh.socket - OpenBSD Secure Shell server socket.

While openssh-server was replaced, sshd was down from 02:15:21 to 02:15:55 UTC. New connections were refused 7 times in a row; logging in again at that moment gives Connection refused. This is about new logins; a session that is already open is a separate matter.

The end of the upgrade log lists the restarted services, a note about the kernel, and a notice from RoamSwitch.

grep -A2 "^Restarting services" /var/tmp/upgrade.log
grep "running kernel" /var/tmp/upgrade.log
cat /var/run/reboot-required /var/run/reboot-required.pkgs
grep -A2 "^RoamSwitch" /var/tmp/upgrade.log

Output

Restarting services...
 /etc/needrestart/restart.d/systemd-manager
 systemctl restart cron.service fail2ban.service multipathd.service packagekit.service polkit.service roamswitch-server.service systemd-journald.service systemd-networkd.service systemd-resolved.service systemd-timesyncd.service systemd-udevd.service udisks2.service
  The currently running kernel version is not the expected kernel version 7.0.0-43-generic.
*** System restart required ***
linux-image-6.8.0-31-generic
linux-base
linux-image-6.8.0-149-generic
linux-base
linux-image-7.0.0-43-generic
linux-base
RoamSwitch: 1 change is not explained by the package update and was not taken into the FIM baseline.
  /etc/cron.d/lab-during-upgrade (new file): it is not a file of any package (it was written while the update command ran, before the packages were unpacked)
Check: sudo roamswitch fim verify. Once you know the changes are legitimate: sudo roamswitch fim update

At the end, needrestart restarted a few services, fail2ban and roamswitch-server among them. The new kernel is not used until the OS reboots. The last three lines say one change was not explained by the package update and was kept out of the tamper baseline. The next section reads it.

Piling on other actions because you cannot get in

If you cannot get in mid-upgrade and force a reboot of the VM or OS, packages are left half replaced. The upgrade inside tmux is still running. Wait, or check from the console with ps whether apt-get is still there. In this VM, sshd was back in about 35 seconds.

#Read the alerts

Read what was reported during and after the upgrade.

Read the upgrade's alerts and the changes no alert shows

Read-only, except one comment line added to /etc/hosts and removed right away (a change to the system).

First, list the alerts raised after the upgrade started.

$R "sudo roamswitch notifications --json | jq -r --arg t \"\$(cat /var/tmp/T0)\" '.[] | select(.timestamp > \$t) | .timestamp[0:19] + \"  \" + .title'"; echo "(end)"

Output

2026-11-12T02:16:27  Critical Path FIM Tampering Detected (1 violation)
(end)

Replacing the kernel, sshd and sudo raised nothing from the tamper monitor or the eBPF guard. While apt runs, the monitor holds its alerts back; when apt finishes, only changes the package update explains are taken into the baseline.

There is still one tampering alert. In the previous lab, right after the upgrade took the dpkg lock, this line was written from the console.

until sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 1; done
echo '# lab dummy, written during the upgrade' | sudo tee /etc/cron.d/lab-during-upgrade

The alert is about that write. Read the body of this file's alert from the console.

sudo roamswitch notifications --json | jq -r '.[] | select(.body | contains("lab-during-upgrade")) | "== " + .timestamp[11:19] + "  " + .title, .body, ""'

Excerpt

== 02:16:27  Critical Path FIM Tampering Detected (1 violation)
/etc/cron.d/lab-during-upgrade: new file
  Process: root / /usr/bin/tee / command: tee /etc/cron.d/lab-during-upgrade (pid 13861)
  No SSH login was found (it is a process of the systemd service tart-guest-agent.service)
  The package update at 2026-11-12 02:16 did not take this change into the baseline: it is not a file of any package (it was written while the update command ran, before the packages were unpacked)

02:16:27: when the upgrade ended, the held alert went out. It names the writer, the console's tee, and says the file belongs to no package, so it was kept out of the baseline
The reason in parentheses shows that the file was written at 02:14:33, while apt-get was still fetching packages, before they were unpacked
Waiting until 02:21 brought no further alert for this file. A change already reported at the end of the upgrade is not reported again by the periodic check

The file also stays in fim verify.

$R 'sudo roamswitch fim verify'

Output

  🔍 Verifying integrity of critical system binaries and configs... (91 files)
  🚨 File modification or tampering detected!
    [new file] /etc/cron.d/lab-during-upgrade
  1 of these was left out of the baseline by the package update at 2026-11-12 02:16, which does not explain it.
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)
  (3 changes look like package upgrades; the daemon takes them into the baseline)

  If these are legitimate changes (e.g. OS upgrade), run 'sudo roamswitch fim update' to baseline.

The replaced set-uid sudo and the others are judged package upgrades by the daemon and taken into its baseline. Files placed during the upgrade, including in places the monitor does not watch, can be found as files newer than the marker that belong to no package.

$R 'sudo find /etc/cron.d /etc/cron.daily /etc/cron.hourly /etc/sudoers.d /etc/pam.d /etc/ssh /etc/systemd/system /etc/profile.d /etc/ld.so.conf.d /root/.ssh /home/*/.ssh -xdev -type f -newer /var/tmp/upgrade-start 2>/dev/null | while read -r f; do dpkg -S "$f" >/dev/null 2>&1 || echo "not from a package: $f"; done'
$R 'sudo dpkg --verify'

Output

not from a package: /etc/cron.d/lab-during-upgrade
??5?????? c /etc/falcoctl/falcoctl.yaml

not from a package: a file placed during the upgrade that belongs to no package. Only the practice file was found
dpkg --verify: package files whose content differs from what was shipped. 5 means the content differs, c marks a configuration file. falcoctl.yaml was rewritten by Falco itself at install time

Once the upgrade is over, the monitor is back to normal. From your PC, add a comment line to /etc/hosts.

$R 'echo "# lab dummy, written after the upgrade" | sudo tee -a /etc/hosts >/dev/null'
sleep 5
sudo roamswitch notifications | sed -n 5,14p
ssh-keygen -lf /var/tmp/lab_key.pub

Output

  2026-11-12T02:17:18.240223310+00:00
  Critical Path FIM Tampering Detected (1 violation)
    Host: ubuntu (IP: 192.168.64.32)
    Integrity violations in critical system files:
    /etc/hosts: modified
      Process: root (logged in as admin) / /usr/bin/tee / command: tee -a /etc/hosts (pid 40234)
      SSH login: the session admin opened from 10.99.0.2:33794 at 2026-11-12 02:17:18 (authentication: publickey, key SHA256:IkcF1OvHdbfC0ydp0366nEBzz03rFyYcTxStBFwdjU8)
    Recommended Action: Run `sudo roamswitch fim verify` to inspect changes.
    Timestamp: 2026-11-12 02:17:18

256 SHA256:IkcF1OvHdbfC0ydp0366nEBzz03rFyYcTxStBFwdjU8 lab-dummy-key (ED25519)

The alert came within 5 seconds. The key fingerprint on the SSH login line matches your key, so this is your own work. Remove the added line and /etc/hosts matches the baseline again.

$R 'sudo sed -i "/lab dummy, written after the upgrade/d" /etc/hosts'

Is an alert after the upgrade expected or not?

Based on what this VM showed:

No alert for the package replacements themselves: expected. Changes the packages explain go into the baseline without an alert
A tamper alert at the end of the upgrade: read it as unexpected. It is a change written during the upgrade by something other than a package. Check the writing process and the SSH login line against your work. If they do not match, investigate
A file that belongs to no package: unless you put it there, read its content. cron, sudoers and systemd units are where persistence gets planted

#Reboot and confirm everything is back

The new kernel only runs after a reboot. After it, confirm not just that the services run, but that detection actually works.

Reboot, confirm the protection works, and clean up

This reboots the OS (a change to the system). The practice host is cleaned up at the end of this lab.

Reboot from your PC.

$R 'sudo systemctl reboot'

The network namespace is gone after the reboot. Recreate outside from the console with the same steps as before, then log in again. Wait about a minute before checking.

$R 'uname -r; systemctl is-active roamswitch-server falco-modern-bpf ssh.socket fail2ban; ls /var/run/reboot-required 2>&1'
$R 'sudo roamswitch status --server | grep -A1 "eBPF Runtime"'
$R 'sudo roamswitch fim verify'
$R 'sudo nft list tables'

Output

7.0.0-43-generic
active
active
active
active
ls: cannot access '/var/run/reboot-required': No such file or directory
  [PASS] eBPF Runtime Security Agent - Active (Falco / Tetragon / RoamSwitch)
         Compliance: CIS - / NIST CSF DE.CM-09
  🔍 Verifying integrity of critical system binaries and configs... (91 files)
  🚨 File modification or tampering detected!
    [new file] /etc/cron.d/lab-during-upgrade
  1 of these was left out of the baseline by the package update at 2026-11-12 02:16, which does not explain it.
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)

  If these are legitimate changes (e.g. OS upgrade), run 'sudo roamswitch fim update' to baseline.
table inet roamswitch_server
table inet roamswitch_metadata_watch
table inet roamswitch_egress_block
table inet roamswitch_portscan
table inet roamswitch_dnstunnel
table inet roamswitch_dev_guard

The host booted the new kernel, and RoamSwitch, Falco and sshd are running. The firewall rules are back. The daemon took the three SUID changes into its baseline, so they no longer show. The cron file placed during the upgrade is still listed after the reboot.

"Running" does not prove the eBPF sensor works on the new kernel. Raise one harmless detection: root's python3 only reads /etc/shadow, it writes nothing.

$R 'sudo python3 -c "open(\"/etc/shadow\").read(); import time; time.sleep(3)"'
sleep 8
sudo roamswitch notifications | sed -n 5,12p

Output

  2026-11-12T02:23:06.838248008+00:00
  eBPF Runtime Incident: Read sensitive file untrusted (Warning)
    Host: ubuntu (IP: 192.168.64.32)
    Rule Triggered: Read sensitive file untrusted
    Severity: Warning
    Process: python3 (PID: 2133)
    Autonomous Containment Action: Logged & Monitored (guard policy: no containment)
    Timestamp: 2026-11-12 02:23:06

Falco's detection reached RoamSwitch on the new kernel. The severity is Warning, so it is recorded and reported only.

Last, clean up the practice host. The older kernels stay in /boot. Remove the practice file and key, check with fim verify, then take the result into the baseline.

ls /boot/vmlinuz-*
sudo rm /etc/cron.d/lab-during-upgrade
sed -i '/lab-dummy-key/d' ~/.ssh/authorized_keys
sudo roamswitch fim verify
sudo roamswitch fim update
sudo ip netns del outside
sudo rm -f /var/tmp/lab_key /var/tmp/lab_key.pub /var/tmp/upgrade-start /var/tmp/T0 /var/tmp/upgrade.log /var/tmp/probe.log /var/tmp/ssh-client.out

Excerpt

/boot/vmlinuz-6.8.0-147-generic
/boot/vmlinuz-6.8.0-149-generic
/boot/vmlinuz-6.8.0-31-generic
/boot/vmlinuz-7.0.0-38-generic
/boot/vmlinuz-7.0.0-43-generic
  🚨 File modification or tampering detected!
    [modified] /home/admin/.ssh/authorized_keys
  If these are legitimate changes (e.g. OS upgrade), run 'sudo roamswitch fim update' to baseline.
  ✓ FIM baseline database successfully updated and saved. (91 entries recorded)

The cron file was never in the baseline, so deleting it took it off the list. What remains is authorized_keys, where you removed the practice key. 6.8.0-31 is the kernel installed while building the practice host; 6.8.0-149 came with this upgrade. To remove Falco, use sudo apt-get remove falco.

Check the same things before and after

Run the fim verify, bans, ssh-allow status and nft list tables you ran before the upgrade again after the reboot. Only what differs needs a closer look.

#What this does not show

What was checked and what was not

Checked on a disposable Ubuntu 24.04 (arm64) VM, by upgrading 60 packages including the kernel (47 upgraded, 13 new):

No major OS upgrade (do-release-upgrade) was done. These results are for updates within one release
Changes to watched files made during the upgrade are reported when it ends, and stay out of the baseline. Places outside the monitor are searched with find against a marker file, plus dpkg --verify. Changes before the marker, or in places the find does not search, are not found this way
How long sshd is down depends on the upgrade. In this VM it was 34 seconds
Configuration files were kept as they were with --force-confold. Handling the difference from new default configuration is not covered
Booting back into the older kernel was not tried. This VM's GRUB hides its menu (GRUB_TIMEOUT=0). On a real host, choosing an older kernel needs a console
This upgrade did not include an update of the Falco package itself

Check yourself

Mid-upgrade, SSH dropped, and logging in again gives Connection refused. The upgrade runs inside tmux. What do you do?

Show the explanation

During an upgrade, replacing openssh-server can stop sshd. The firewall is not the cause. The upgrade inside tmux keeps running, and sshd returns when it finishes.

Check yourself

Before the upgrade, fim verify shows a change to /etc/pam.d/common-auth that you do not recognise. What do you do?

Show the explanation

An upgrade replaces package files. If a tampered file is replaced by the same upgrade, its content matches the package's record and joins the baseline, and the trace of the change is gone. Investigate before the upgrade.

#Next steps

The files the tamper monitor watches, and how to read its alerts, are covered in Persistence and tampering
Getting out of a lockout by the brute-force guard or the allow-list is in SSH brute force, and when you lock yourself out
Updating RoamSwitch itself and reading doctor are in Updates, threat data and uninstalling

Features covered

Related techniques (ATT&CK): T1098.004 T1548.003 T1548.001

What was verified

Verified on 2026-11-12. Target: Linux 1.17.39. On a disposable Ubuntu 24.04 VM with the published 1.17.39, moved back to the previous kernel, then ran apt full-upgrade inside tmux, the disconnect, the reboot and the cleanup. English output was taken on the same VM by switching LANG (and /etc/default/locale for the SSH side).

Command output varies with the product version and the environment. If yours differs, check the version first.