Logs were deleted or rewritten

This rewrites auth.log (it changes the machine). Back it up first. Replace admin with your own account name.

sudo cp -p /var/log/auth.log /root/auth.log.bak
ssh-keygen -q -t ed25519 -N "" -f ~/.ssh/lab_key
cat ~/.ssh/lab_key.pub >> ~/.ssh/authorized_keys
ssh -o StrictHostKeyChecking=no -i ~/.ssh/lab_key admin@127.0.0.1 true
sleep 35
ssh -o StrictHostKeyChecking=no -i ~/.ssh/lab_key admin@127.0.0.1 \
  'sudo sed -i "/Accepted publickey/d" /var/log/auth.log'
sleep 35
sudo roamswitch notifications | grep -A8 'Log Wiping' | head -9

  Possible Log Wiping Detected (1)
    Host: ubuntu (IP: 192.168.64.30)
    These logs changed in a way log rotation does not explain:
    /var/log/auth.log was replaced by a different file (another inode); rewriting it with sed -i or the like does this
    An attacker may have erased their traces.
    Check: look at recent logins with `last` and `sudo journalctl -u ssh`, and at changes to critical files and SUID files with `sudo roamswitch fim verify`.
    Timestamp: 2026-11-12 02:01:49
    The process that made the change cannot be identified (it was found by comparing the state of the files periodically)

Three things come from the alert.

• Which file changed, and how. Here auth.log was replaced by a different file with another inode, because sed -i writes a new file and swaps it in for the old one
• It is judged a change that log rotation does not explain. A legitimate rotation is excluded
• As the last line says, this alert cannot tell you who erased it, because it is found by comparing the state of the file (size and inode) every 30 seconds

When you added the key to authorized_keys while preparing, Critical system file tamper monitoring also sent one "Critical Path FIM Tampering Detected" alert. It names the writing process and the SSH login (or, when no login is found, the service the process belongs to), so you can confirm it was your own change.

The next section fills in that "who".

This only reads records. It changes nothing. With no argument events search returns the most recent records; with --text it returns only records that contain that string.

sudo roamswitch events search --text /var/log/auth.log

2026-11-12T02:01:25.937Z pid=1643 ppid=1642 root /usr/bin/sed -i /Accepted publickey/d /var/log/auth.log
2026-11-12T02:01:25.852Z pid=1642 ppid=1641 admin /usr/bin/sudo sed -i /Accepted publickey/d /var/log/auth.log
2026-11-12T02:01:25.852Z pid=1642 ppid=1641 [euid 1000→0] admin /usr/bin/sudo sed -i /Accepted publickey/d /var/log/auth.log
2026-11-12T02:01:25.851Z pid=1642 ppid=1641 admin /usr/bin/bash -c sudo sed -i "/Accepted publickey/d" /var/log/auth.log
2026-11-12T02:01:25.540Z pid=1578 ppid=1154 admin /usr/bin/ssh -o StrictHostKeyChecking=no -i /home/admin/.ssh/lab_key admin@127.0.0.1 sudo sed -i "/Accepted publickey/d" /var/log/auth.log

What touched auth.log was root's sed -i. The exact command that removed the lines is still there. Its parent, PID 1642, is admin's bash -c turned into sudo; the [euid 1000→0] line is where that sudo became root. The last line is the ssh client that came in and passed the command. Pass the sed PID to events tree.

sudo roamswitch events tree 1643

1 /usr/lib/systemd/systemd
1266 /usr/sbin/sshd
1579 /usr/sbin/sshd
1641 /usr/sbin/sshd
1642 /usr/bin/sudo
            1643 /usr/bin/sed -i /Accepted publickey/d /var/log/auth.log

Top to bottom is parent to child. sshd is the entry point, then sudo, then sed, which rewrote auth.log. The "who" the alert left blank is answered by this chain. When the intruder came from outside, the top sshd points to the session they used.

This only reads. It changes nothing.

sudo grep -c 'Accepted publickey' /var/log/auth.log
sudo journalctl -u ssh --no-pager | grep 'Accepted publickey' | tail -1

0
Nov 12 02:01:25 ubuntu sshd[1579]: Accepted publickey for admin from 127.0.0.1 port 39872 ssh2: ED25519 SHA256:nY+QrblgM6n3y9ixtmpX4DxbYuq5d70oYaxc0sB8g9g

It is gone from auth.log, but the same login survives in the journal, because sshd writes there too. sshd[1579] is the same sshd PID that events tree showed. This is where you learn where the intruder came from. Next, see the erasure itself on a timeline with Incident timeline.

sudo roamswitch timeline | grep -B1 -A2 'log_replaced'

  2026-11-12T02:01:49.314749461+00:00  [FIM]  recorded only
    Log tampering (log_replaced): /var/log/auth.log
    MITRE ATT&CK: T1685.006 (Disable or Modify Tools: Clear Linux or Mac System Logs)
    action taken: notified

On the timeline the erasure appears as a technique (ATT&CK T1685.006). "recorded only" and "notified" mean it only alerted: no process was stopped and nothing was isolated.

This reads, and writes to a file. It does not change the logs being watched.

sudo roamswitch events verify

✓ Hash chain intact (1 segment). No tampering detected (tamper-evident, not tamper-proof).
  The 43 records after the newest checkpoint are not covered until the next checkpoint.

The records are linked by hashes. The second line appears only when there are records after the newest checkpoint. If a record in the middle is erased or rewritten, the chain breaks and it shows. Next, export the records about auth.log as JSON Lines and keep them as evidence.

sudo roamswitch events export --text auth.log > /tmp/exec-auth.jsonl
wc -l /tmp/exec-auth.jsonl

19 /tmp/exec-auth.jsonl

One line is one record. --text auth.log picks records whose command line contains auth.log, so your own export command is included too. Copy it to a separate, safe place as evidence for the investigation.

This changes a setting (it changes the machine). Here you run a receiver on the same machine to see how it arrives. In production you point it at a syslog server or SIEM on another machine.

sudo mkdir -p /var/log/roamswitch
sudo roamswitch server config set forward_file_path /var/log/roamswitch/forward.jsonl
sudo roamswitch server config set forward_enabled true
sudo roamswitch server restart
sleep 5
sudo roamswitch forward status
sudo roamswitch forward test

Event forwarding (SIEM/syslog/webhook) - opt-in, default OFF
  enabled: yes
  running: submitted=0 dropped(oldest)=0 retry queue=0 items / 0 bytes
  - file: ok=0 failed=0 last error=-
  OK   file

Forwarding sends only detections made after it was turned on, so the detection from the first lab is not at the destination yet. Run the same sed -i from the first lab once more.

A second erasure of the same file is not reported at once. Erasures within 5 minutes of the previous alert are held, then reported together as one alert when the 5 minutes are up. The hold survives the server restart in between. So wait until a line reaches the destination.

ssh -o StrictHostKeyChecking=no -i ~/.ssh/lab_key admin@127.0.0.1 \
  'sudo sed -i "/Accepted publickey/d" /var/log/auth.log'
until sudo grep -q 'log_replaced' /var/log/roamswitch/forward.jsonl; do sleep 10; done
sudo grep 'log_replaced' /var/log/roamswitch/forward.jsonl | tail -1

{"host":"ubuntu","event_id":"18e72a0cfc48db29-2084","ts":"2026-11-12T02:06:51.811Z","source":"timeline.fim","severity":"critical","rule":"T1685.006","action":"notified","pid":null,"proc":null,"summary":"Log tampering (log_replaced): /var/log/auth.log"}

About 5 minutes after the first alert (02:01:49), at 02:06:51, the second erasure was detected and the destination kept one record. The alert at that time adds "(it happened 1 more time to the same file since the previous notification; the last change is shown)". The summary of a timeline.fim line is in English whatever the screen language. The alert itself also reaches the destination as one line ("source":"notification"), written in the server's language. If the destination is another machine, erasing the logs here cannot erase that copy. Add Security log audit, which counts events from the journal, to your regular monitoring too, because the journal survives separately when auth.log is rewritten.

sudo roamswitch audit-logs 1 | head -14

=====================================================
  📜 Security Log Audit (last 1h)
=====================================================

  • sudo auth failure lines:   0 lines
  • SSH failure lines:         0 lines
  • Firewall drop lines:       0 lines
  • AppArmor/LSM denial lines: 0 lines
  • Malware detection lines:   0 lines
  • USB connection lines:      0 lines
    These are numbers of log lines, not of attempts. One failure can be two or more lines (an SSH login to a user that does not exist logs "Invalid user" and "Failed password"). SSH lines are failed passwords, users that do not exist, and repeated disconnects from the same source. Successful logins are not counted.

  ⚠ Only the latest 500 log lines were examined. Earlier lines are not included.

The counts are log lines. Successful logins are not counted, so the logins in this lesson leave them at 0.

Finally, restore what the labs changed. Writing the copy back makes auth.log smaller, so that is also reported as possible log wiping (with Falco running, Clear Log Activities appears too). Within 5 minutes of the previous alert, it arrives when the 5 minutes are up, as "1 more time". These alerts come from your own change.

sudo cp /root/auth.log.bak /var/log/auth.log
sudo sed -i '/lab_key/d' ~/.ssh/authorized_keys
rm -f ~/.ssh/lab_key ~/.ssh/lab_key.pub
sudo roamswitch server config set forward_enabled false
sudo roamswitch server restart