What an intruder leaves behind | Learn Defense with RoamSwitch

#What an intruder leaves behind

An intruder who gets into a server leaves something that lets them back in. With root, they also make sure to keep it. Last, they clear the logs to hide their tracks. The common moves, and the Server Edition features that watch for them, are these.

Move	Example	Watched by
Add a key to come back without a password	Write their public key into `/root/.ssh/authorized_keys`	Critical system file tamper monitoring
Change who may use root	Add a file to `/etc/sudoers.d/`	Same
Register a schedule or a service	Place a file in `/etc/cron.d/` or `/etc/systemd/system/`	Same
Leave a program that runs as root	Set the SUID bit on their own program	SUID/SGID change monitoring
Erase the traces	Empty `/var/log/auth.log`	Log wiping monitoring

All of these happen after root is lost. So the features in this module notify and do not block: the other side already has root. The last chapter covers the settings that make root harder to get (Kernel hardening and Check of kernel modules that open privilege-escalation paths).

Practice in a disposable environment

The labs in this module change critical files and logs on purpose. Try them in a disposable VM. The lab VM's clock is UTC.

#Find changes to critical files

Critical system file tamper monitoring records the hashes of sudoers, PAM, the sshd settings, cron, systemd units, every account's authorized_keys and more. Writes, and new files in the watched folders, are seen as they happen (fanotify). A periodic scan compares everything again.

This changes a critical file (it changes the machine). You undo it at the end.

In this lab, the login was an SSH session from inside the same VM to 127.0.0.1. Replace admin with your own account name. The key you add is an invalid dummy that only has the right shape.

ssh-keygen -q -t ed25519 -N "" -f ~/.ssh/lab_key
cat ~/.ssh/lab_key.pub >> ~/.ssh/authorized_keys
ssh -o StrictHostKeyChecking=accept-new -i ~/.ssh/lab_key admin@127.0.0.1 \
  'echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEXAMPLE lab@example" | sudo tee -a /root/.ssh/authorized_keys >/dev/null'
sleep 5
sudo roamswitch notifications | grep -A8 FIM | head -9

Excerpt of the output

  Critical Path FIM Tampering Detected (1 violation)
    Host: ubuntu (IP: 192.168.64.8)
    Integrity violations in critical system files:
    /root/.ssh/authorized_keys: modified
      Process: root (logged in as admin) / /usr/bin/tee / command: tee -a /root/.ssh/authorized_keys (pid 29125)
      SSH login: the session admin opened from 127.0.0.1:54970 at 2026-11-12 01:41:32 (authentication: publickey, key SHA256:rkBIL270U06lSAVL6PeFRbgM29moyiChfjIWkAVaSBA)
    Recommended Action: Run `sudo roamswitch fim verify` to inspect changes.
    Timestamp: 2026-11-12 01:41:33

The notification came right after the write. The kind is modified. Look at three things.

Process: root's tee wrote the file. The account that logged in (admin) is shown too
SSH login: where the session came from (127.0.0.1:54970), when, and with which key. When someone gets in from outside, their address is here
The second command: adding a key to your own authorized_keys raised a separate notification (left out of the excerpt). Every account's keys are watched

The login details come from fim_intrusion_context_enabled (default true). It only reads /proc, the exec recorder and logind, and sends nothing anywhere (per the notes in the example settings file).

When a notification arrives, use fim verify to see what changed. After you undo it, check again.

sudo roamswitch fim verify
sudo sed -i '/lab@example/d' /root/.ssh/authorized_keys
sudo roamswitch fim verify
sudo roamswitch fim update

Excerpt of the output

  🔍 Verifying integrity of critical system binaries and configs... (90 files)
  🚨 File modification or tampering detected!
    [modified] /root/.ssh/authorized_keys
    [modified] /home/admin/.ssh/authorized_keys
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)

  If these are legitimate changes (e.g. OS upgrade), run 'sudo roamswitch fim update' to baseline.
  🔍 Verifying integrity of critical system binaries and configs... (90 files)
  🚨 File modification or tampering detected!
    [modified] /home/admin/.ssh/authorized_keys
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)

  If these are legitimate changes (e.g. OS upgrade), run 'sudo roamswitch fim update' to baseline.
  📝 Generating baseline database from current critical files...
  ✓ FIM baseline database successfully updated and saved. (90 entries recorded)
  ✓ SUID / SGID baseline updated. (20 entries recorded)

Once the root key line was removed, /root/.ssh/authorized_keys dropped off the list: its contents matched the baseline again. The remaining /home/admin/.ssh/authorized_keys held the key you added yourself, so fim update accepted it.

Running `fim update` before you check

fim update takes everything as it is now as the trusted state. If a key the intruder added is still there, it becomes trusted and is never reported again. Go through the fim verify list line by line, and run it only when nothing but what you want to keep is left.

Newly placed files

Placing a new cron or unit file is caught at once too. A file that appears in a watched folder such as /etc/cron.d, /etc/sudoers.d or /etc/systemd/system is reported right away.

Place a schedule file in two ways

This places files (it changes the machine). Log in over SSH with the key from the previous lab. Write one file with tee, and make the other in /tmp and move it in with mv. Both only run /usr/bin/true, which does nothing.

ssh -i ~/.ssh/lab_key admin@127.0.0.1 \
  'echo "*/5 * * * * root /usr/bin/true" | sudo tee /etc/cron.d/lab-persist >/dev/null'
ssh -i ~/.ssh/lab_key admin@127.0.0.1 \
  'echo "*/5 * * * * root /usr/bin/true" > /tmp/lab-mv && sudo mv /tmp/lab-mv /etc/cron.d/lab-mv'
sleep 5
sudo roamswitch notifications | grep -A6 FIM | head -14

Excerpt of the output

  Critical Path FIM Tampering Detected (1 violation)
    Host: ubuntu (IP: 192.168.64.8)
    Integrity violations in critical system files:
    /etc/cron.d/lab-mv: new file
      The login path could not be determined (it was moved into place (mv) from another name, so the process that wrote it cannot be identified)
    Recommended Action: Run `sudo roamswitch fim verify` to inspect changes.
    Timestamp: 2026-11-12 01:41:08
--
  Critical Path FIM Tampering Detected (1 violation)
    Host: ubuntu (IP: 192.168.64.8)
    Integrity violations in critical system files:
    /etc/cron.d/lab-persist: new file
      Process: root (logged in as admin) / /usr/bin/tee / command: tee /etc/cron.d/lab-persist (pid 28731)
      SSH login: the session admin opened from 127.0.0.1:46076 at 2026-11-12 01:41:08 (authentication: publickey, key SHA256:rkBIL270U06lSAVL6PeFRbgM29moyiChfjIWkAVaSBA)

Both showed up as new file about a second after they were placed. For the file written with tee, the process and the SSH login are known. For the one moved in with mv, the process that wrote it is not known. To find who placed it, compare the file's times with the login records in last and journalctl.

On the same VM, lab-persist was deleted and put back with the same content. Putting it back raised another new file notification, so delete-and-replace is not missed.

Undo it. Delete the files and check.

sudo rm /etc/cron.d/lab-persist /etc/cron.d/lab-mv
sudo roamswitch fim verify

Output

  🔍 Verifying integrity of critical system binaries and configs... (90 files)
  ✓ No tampering detected. All files verified healthy. (90 files monitored)
  🔍 Checking the SUID / SGID file inventory...
  ✓ No SUID / SGID file was added or changed. (20 files checked)

#Find files with the SUID bit

A program with the SUID bit runs with its owner's rights, not those of whoever runs it. If root owns it, anyone who runs it is root. An intruder with root leaves one as a private way back. SUID/SGID change monitoring keeps the list of SUID and SGID files as a baseline and reports additions and changes.

Place a program with the SUID bit

This creates a file that runs as root (it changes the machine). You delete it at the end. It is a copy of id, which only prints your IDs.

sudo cp /usr/bin/id /usr/local/bin/lab-id
sudo chmod u+s /usr/local/bin/lab-id
/usr/local/bin/lab-id
sleep 5
sudo roamswitch notifications | grep -A10 SUID | head -11
sudo roamswitch fim verify

Excerpt of the output

uid=1000(admin) gid=109(admin) euid=0(root) groups=109(admin)
  SUID/SGID File Added or Changed (1 file)
    Host: ubuntu (IP: 192.168.64.8)
    These executables with the SUID or SGID bit changed:
    /usr/local/bin/lab-id now has SUID/SGID (new file or bit added later; mode 4755, owner UID 0; owned by no package)
    A SUID executable runs with its owner's rights (usually root), not those of whoever runs it. Attackers use this to keep root.
    Check: look at the owner and modification time with `ls -l` and at recent logins with `last`.
    If you know the change is legitimate: run `sudo roamswitch fim update` to accept the current state.
    Timestamp: 2026-11-12 01:41:16
    Process: root / /usr/bin/chmod / command: chmod u+s /usr/local/bin/lab-id (pid 28858)
    Parent processes: sudo ← dash ← tart-guest-agent ← systemd / Service: tart-guest-agent.service
  🔍 Verifying integrity of critical system binaries and configs... (90 files)
  ✓ No tampering detected. All files verified healthy. (90 files monitored)
  🔍 Checking the SUID / SGID file inventory...
  🚨 SUID / SGID files were added or changed!
    [SUID/SGID added] /usr/local/bin/lab-id (4755)

Look at the first line. The ordinary account admin ran it, and the result is euid=0(root). This is why SUID files are used to stay in.

The notification shows the process that ran chmod and its parents, because mode changes are seen as they happen (fanotify). The whole list is also rebuilt and compared every suid_watch_interval_secs (default 600 seconds). "Owned by no package" is an important clue. A change that a package upgrade explains is taken into the baseline without a notification (per the notes in the example settings file).

Undo it.

sudo rm /usr/local/bin/lab-id
sudo roamswitch fim verify | tail -2

Output

  ✓ No SUID / SGID file was added or changed. (20 files checked)

An SUID file you know to be legitimate goes into the baseline with fim update. Until then, fim verify keeps reporting it as a failure.

#Find wiped logs

Logs change all the time as lines are added. So FIM, which compares hashes, cannot tell when one is wiped. Log wiping monitoring looks at auth.log, syslog, audit.log, wtmp and others every 30 seconds and reports a log that got smaller, disappeared or was replaced by a different file. Changes that log rotation explains are left out.

Empty the authentication log

This wipes a log (it changes the machine). Make a copy first and put it back at the end.

sudo cp -p /var/log/auth.log /root/auth.log.bak
sudo truncate -s 0 /var/log/auth.log
sleep 35
sudo roamswitch notifications | grep -A8 'Log Wiping' | head -9

Output

  Possible Log Wiping Detected (1)
    Host: ubuntu (IP: 192.168.64.8)
    These logs changed in a way log rotation does not explain:
    /var/log/auth.log became smaller (5930 bytes → 99 bytes)
    An attacker may have erased their traces.
    Check: look at recent logins with `last` and `sudo journalctl -u ssh`, and at changes to critical files and SUID files with `sudo roamswitch fim verify`.
    Timestamp: 2026-11-12 01:55:38
    The process that made the change cannot be identified (it was found by comparing the state of the files periodically)

The notification came within 30 seconds of the wipe. The 99 bytes were written after it was emptied. As the last line says, who wiped it is not known: it is found by comparing file sizes periodically.

Put the copy back.

sudo cp /root/auth.log.bak /var/log/auth.log
sudo rm /root/auth.log.bak

In the same VM, other ways of wiping and a legitimate task were tried too.

Deleting only some lines with sudo sed -i '/127.0.0.1/d' /var/log/auth.log was reported as replaced by a different file (another inode) and smaller, because sed -i writes a new file and swaps it in
When the same file changed again within 5 minutes of the last notification, it was not reported at once. After the 5 minutes, one notification came with the count, saying it happened 1 more time to the same file. It still came when sudo roamswitch server restart was run while it was held
Rotating with sudo logrotate -f /etc/logrotate.conf left a new, small auth.log (99 bytes 40 seconds later), but raised no notification. The old contents were in auth.log.1, which rotation explains

When logs were wiped

Clues remain after a wipe. As the notification's "Check" line says, look at last (the login records read from wtmp) and journalctl. Then run sudo roamswitch fim verify to see whether keys or SUID files were added. A way back in is often placed before the logs are wiped.

#Fewer footholds

The features so far report after root is lost. Kernel hardening works one step earlier and removes footholds for raising privileges. Each time the Server Edition daemon starts, it writes kernel settings (sysctl).

Look at the kernel settings and what they do

Run this as an ordinary account (admin). It changes nothing.

sysctl kernel.yama.ptrace_scope kernel.unprivileged_userns_clone kernel.unprivileged_bpf_disabled
strace -o /dev/null true 2>&1; echo "strace: $?"
unshare -U true 2>&1; echo "unshare: $?"
sudo roamswitch server config get harden_userns_enabled

Output

kernel.yama.ptrace_scope = 2
kernel.unprivileged_userns_clone = 0
kernel.unprivileged_bpf_disabled = 1
strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Operation not permitted
strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted
strace: 1
unshare: unshare failed: Operation not permitted
unshare: 1
true

The three settings mean the following.

Setting	Value	Effect
`kernel.yama.ptrace_scope`	2	Only administrators may use ptrace, which reads into other processes. An ordinary account cannot trace even its own child
`kernel.unprivileged_userns_clone`	0	An ordinary account cannot create a user namespace
`kernel.unprivileged_bpf_disabled`	1	An ordinary account cannot load eBPF programs

For comparison, a freshly booted VM where the daemon had never run was checked too. The values were ptrace_scope 1, unprivileged_userns_clone 1 and unprivileged_bpf_disabled 2 (the Ubuntu 24.04 defaults). strace -o /dev/null true and unshare -U true both passed with exit code 0. RoamSwitch puts no file in /etc/sysctl.d/; it writes the values each time it starts.

Whether to turn off `harden_userns_enabled`

Denying user namespaces breaks containers that run without root (Podman rootless and the like) and apps that run in a sandbox (per the notes in the example settings file). If you use them, run sudo roamswitch server config set harden_userns_enabled false.

Turning it off does not undo it right away. Setting false prints "The limit on unprivileged user namespaces is lifted when the OS reboots."

In the same VM, after server restart, unprivileged_userns_clone stayed 0 and unshare -U true still failed. After the OS was restarted, it was 1 and unshare passed. Setting true again and running server restart put it back to 0 at once.

As the notes in the settings file say, the real fix is to update to a patched kernel.

Check of kernel modules that open privilege-escalation paths checks whether this machine can load the kernel modules that 2026's local privilege escalations went through.

Check risky modules and stop one from loading

Checking changes nothing. --apply and --remove write to /etc/modprobe.d/, so they change the machine.

roamswitch kernel-modules

Excerpt of the output

  kernel 7.0.0-42-generic
  Unprivileged user namespaces: unavailable (the path to esp4, esp6 and act_pedit is closed)
  
  algif_aead  blocked  [Copy Fail / CVE-2026-31431]
  esp4        unused, loaded on demand  [Dirty Frag / Fragnesia / CVE-2026-43284]
  esp6        unused, loaded on demand  [Dirty Frag / Fragnesia / CVE-2026-43284]
  rxrpc       unused, loaded on demand  [Dirty Frag / DirtyDecrypt / CVE-2026-43500]
  act_pedit   unused, loaded on demand  [act_pedit page-cache write / CVE-2026-46331]

The second line ties in with the user-namespace setting from the previous lab. Namespaces are unavailable, so the path to esp4, esp6 and act_pedit is judged closed. In this VM, algif_aead was already blocked by Ubuntu's own /etc/modprobe.d/disable-algif_aead.conf.

Stop the unused esp4 from loading, and check that it really cannot be loaded.

sudo roamswitch kernel-modules --apply esp4
cat /etc/modprobe.d/roamswitch-kernel-module-surface.conf
sudo modprobe esp4 2>&1; echo "modprobe: $?"

Excerpt of the output

Wrote: /etc/modprobe.d/roamswitch-kernel-module-surface.conf (esp4)
  esp4        blocked  [Dirty Frag / Fragnesia / CVE-2026-43284]
# Written by RoamSwitch (roamswitch kernel-modules --apply). Stops these kernel modules from
# being loaded on demand. Undo: roamswitch kernel-modules --remove (or delete this file).
install esp4 /bin/false
modprobe: ERROR: ../libkmod/libkmod-module.c:1084 command_do() Error running install command '/bin/false' for module esp4: retcode 1
modprobe: ERROR: could not insert 'esp4': Invalid argument
modprobe: 1

Undo it. The rule file is deleted and esp4 can be loaded again.

sudo roamswitch kernel-modules --remove
roamswitch kernel-modules | grep '^  esp4'

Excerpt of the output

Removed the rule file.
  esp4        unused, loaded on demand  [Dirty Frag / Fragnesia / CVE-2026-43284]

esp4 and esp6 in use by an IPsec VPN (strongSwan and the like) are left out automatically (per the note in the command's output). The command does not judge whether the kernel is already patched.

#What it does not see

What was and was not confirmed

Within what was tried in a disposable Ubuntu 24.04 VM:

Per-account crontabs and ~/.bashrc are outside the Server FIM. As admin, a crontab - entry and a line added to ~/.bashrc: fim verify said all files were healthy, and no FIM notification was added. What it watches is the system-wide places such as /etc/cron.d/
For a file moved in with mv, the process that wrote it is not known. That it was placed in a watched folder is seen at once. Who wrote it has to be traced from the file's times and the login records
For a wiped log, the process is not known either. The comparison runs every 30 seconds, so a log other than wtmp and btmp that is emptied and refilled to its old size in between is not seen (confirmed in the code's notes)
It notifies and does not stop anything. After the notifications in this module, no process or network was isolated. What to do after root is lost is up to a person
The kernel settings do not stop every privilege escalation. In this VM, sudo strace -o /dev/null true still passed. They do not apply to a process with administrator rights. The real fix is a kernel update

Check yourself

sudo roamswitch fim verify shows [new file] /etc/cron.d/backup-sync. You do not recognize it. What do you do first?

Show the explanation

fim update takes everything as it is now as trusted. The intruder's file becomes trusted and is never reported again. If the notification shows the process that wrote it, follow that. If it was moved in with mv and shows none, trace who placed it and when from its contents and times. Delete it only after checking for other ways back in (keys, SUID files).

Check yourself

To use rootless containers, you set harden_userns_enabled to false and ran sudo roamswitch server restart. What happens to unshare -U true?

Show the explanation

The daemon writes the deny value when it starts. With false it no longer writes it, but it does not undo a value it already wrote. In the same VM, it passed after the OS was restarted.

#What to learn next

Forwarding FIM notifications to a SIEM or syslog is covered in Notifications and event forwarding
How to investigate after a containment fired is in A containment fired: the first hour
What to do when logs were wiped is covered in "Logs were deleted or rewritten" (coming soon)
Guarding SSH, the usual way in for adding a key, is covered in Configuring SSH protection
Every setting is in the Linux Server operations manual

Features covered

Related techniques (ATT&CK): T1098.004 T1548.003 T1548.001 T1685.006 T1055

What was verified

Verified on 2026-11-12. Target: Linux 1.17.39. On a disposable Ubuntu 24.04 VM with the published 1.17.39, keys, cron files, SUID and log wipes were done through SSH logins to the same VM, including delete-and-replace, a repeat within 5 minutes with a daemon restart while it was held, sed -i and logrotate, the kernel settings and modules, and an OS reboot after turning harden_userns_enabled off, in Japanese and English.

Command output varies with the product version and the environment. If yours differs, check the version first.