Notifications and event forwarding

This edits the settings file and restarts the service (it changes the machine). Try it in a practice environment. The steps to undo it are in a later section.

Start two receivers. The webhook one listens on port 8099 (TCP) and records the signature header and the body. The syslog one listens on port 5514 (UDP).

cat > /tmp/recv.py <<'PY'
import http.server
class H(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get('content-length', 0))
        body = self.rfile.read(n)
        sig = self.headers.get('X-RoamSwitch-Signature', '')
        with open('/tmp/fwd.log', 'a') as f:
            f.write(sig + '\n' + body.decode() + '\n')
        open('/tmp/fwd.body', 'wb').write(body)
        open('/tmp/fwd.sig', 'w').write(sig)
        self.send_response(200); self.end_headers()
    def log_message(self, *a): pass
http.server.HTTPServer(('127.0.0.1', 8099), H).serve_forever()
PY
cat > /tmp/syslog.py <<'PY'
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('127.0.0.1', 5514))
while True:
    data, _ = s.recvfrom(65535)
    with open('/tmp/syslog.log', 'a') as f:
        f.write(data.decode(errors='replace') + '\n')
PY
nohup python3 /tmp/recv.py >/dev/null 2>&1 &
nohup python3 /tmp/syslog.py >/dev/null 2>&1 &

Put the secret that signs the webhook body in a file only root can read. labsecret123 is a practice dummy. The receivers are on the same machine (127.0.0.1). Sending to internal addresses is refused by default, so allow it by writing 127.0.0.1/32 in forward_webhook_allow_addresses. This works the same on a server updated from an earlier version (the migration of the old default false is in the decision section). Alert text follows language, so set it to en. Then set the destinations, restart, and send a test event.

sudo sh -c 'umask 077; echo labsecret123 > /etc/roamswitch/fwd.secret'
sudo roamswitch server config set language en
sudo roamswitch server config set forward_webhook_url http://127.0.0.1:8099/events
sudo roamswitch server config set forward_webhook_secret_file /etc/roamswitch/fwd.secret
sudo roamswitch server config set forward_syslog_url udp://127.0.0.1:5514
sudo roamswitch server config set forward_file_path /var/log/roamswitch/events.jsonl
sudo roamswitch server config set forward_webhook_allow_addresses 127.0.0.1/32
sudo roamswitch server config set forward_enabled true
sudo roamswitch server restart
sleep 5
sudo roamswitch forward status
sudo roamswitch forward test

Event forwarding (SIEM/syslog/webhook) - opt-in, default OFF
  enabled: yes
  running: submitted=0 dropped(oldest)=0 retry queue=0 items / 0 bytes
  - syslog_udp: ok=0 failed=0 last error=-
  - file: ok=0 failed=0 last error=-
  - webhook: ok=0 failed=0 last error=-
  OK   syslog_udp
  OK   file
  OK   webhook

The forward_ keys take effect only after a restart. When forward status lists all three destinations, the daemon has loaded them. The test event reports OK or FAIL for each destination.

Right after a restart, forward status can say the daemon has not started forwarding yet. In this check it said "running" right after the restart both times. The sleep 5 is there to be safe.

Look at what reached the three places.

cat /tmp/syslog.log
sudo cat /var/log/roamswitch/events.jsonl
cat /tmp/fwd.log

<38>1 2026-11-12T01:54:30.070Z ubuntu roamswitch 1442 forward_test [roamswitch@32473 event_id="1a1cd76b036-5a2-0" rule="FWD-TEST" action="none" severity="info"] RoamSwitch event-forwarding test message
{"host":"ubuntu","event_id":"1a1cd76b036-5a2-0","ts":"2026-11-12T01:54:30.070Z","source":"forward_test","severity":"info","rule":"FWD-TEST","action":"none","pid":null,"proc":null,"summary":"RoamSwitch event-forwarding test message"}
sha256=845116342ad6a9d9e925a66a0f6dd79bea84c435d78a6e52f13c7abfeaa8f7e1
{"host":"ubuntu","event_id":"1a1cd76b036-5a2-0","ts":"2026-11-12T01:54:30.070Z","source":"forward_test","severity":"info","rule":"FWD-TEST","action":"none","pid":null,"proc":null,"summary":"RoamSwitch event-forwarding test message"}

One event with the same event_id reached all three. Read it like this.

• syslog is RFC 5424. The leading <38> is facility 4 (auth) × 8 + severity 6 (info)
• The file and the webhook carry the same JSON: host, ID, time, source, severity, rule, action, PID, process name and summary
• ts is UTC (the trailing Z). 01:54:30Z is 10:54:30 on November 12 in Japan time
• The file /var/log/roamswitch/events.jsonl is -rw-------, readable by root only. To let a SIEM agent that is not root read it, name a group in forward_file_group

The webhook receiver can use the signature to check that a body came from RoamSwitch. The X-RoamSwitch-Signature header is sha256= followed by the HMAC-SHA256 of the body.

cat > /tmp/verify.py <<'PY'
import hmac, hashlib
key = open('/etc/roamswitch/fwd.secret', 'rb').read().strip()
body = open('/tmp/fwd.body', 'rb').read()
got = open('/tmp/fwd.sig').read().strip()
want = 'sha256=' + hmac.new(key, body, hashlib.sha256).hexdigest()
print('match' if hmac.compare_digest(got, want) else 'MISMATCH')
PY
sudo python3 /tmp/verify.py
python3 -c "import hmac,hashlib;print(hmac.compare_digest(open('/tmp/fwd.sig').read().strip(),'sha256='+hmac.new(b'wrongsecret',open('/tmp/fwd.body','rb').read(),hashlib.sha256).hexdigest()))"

match
False

The right secret matches and a wrong one does not. On the receiving side, compute over the exact bytes of the body you received. If you parse the JSON and write it out again, spacing or order can change and the check fails. Compare with a constant-time function such as hmac.compare_digest.

Add one comment line to /root/.ssh/authorized_keys (it changes the machine). A comment line does not work as a login key. You put the file back within this lesson.

: > /tmp/syslog.log
sudo sh -c 'echo "# lab comment" >> /root/.ssh/authorized_keys'
sleep 10
cat /tmp/syslog.log

<36>1 2026-11-12T01:54:31.685Z ubuntu roamswitch 1209 timeline.fim [roamswitch@32473 event_id="18e72960a9fffc06-1209" rule="" action="notified" severity="medium" pid="1461" proc="dash"] /root/.ssh/authorized_keys: modified (no SSH login found: service tart-guest-agent.service)
<37>1 2026-11-12T01:54:31.731Z ubuntu roamswitch 1209 notification [roamswitch@32473 event_id="1a1cd76b6b3-4b9-1" rule="" action="notify" severity="low"] Critical Path FIM Tampering Detected (1 violation): Host: ubuntu (IP: 192.168.64.24) / Integrity violations in critical system files: / /root/.ssh/authorized_keys: modified / Process: root / /usr/bin/dash / command: sh -c echo "# lab comment" >> /root/.ssh/authorized_keys (pid 1461) (The process had already exited, so this comes from the exec record made at 2026-11-12 01:54:31.) / No SSH login was found (it is a process of the systemd service tart-guest-agent.service) / Recommended Action: Run `sudo roamswitch fim verify` to inspect changes. / Timestamp: 2026-11-12 01:54:31

One change produced two events.

• timeline.fim: the detection itself. Its severity is medium, and it names the writing process (dash, PID 1461). The summary is a fixed English text
• notification: a copy of the alert that was raised. Alerts carry no severity, so they are always sent as low (a note in the code). The summary is the whole alert text, with / between its lines, in the language of the server's language setting

Put the file back. This writes the old contents into the same file instead of replacing it. The contents match the baseline again, so no new detection comes out. Adding the same line again after that brought a new detection and alert copy, with no restart.

sudo sh -c 'grep -v "^# lab comment$" /root/.ssh/authorized_keys > /tmp/ak; cat /tmp/ak > /root/.ssh/authorized_keys'

Next, raise the lowest severity that is sent, forward_min_severity (default low), to medium, and make the same change.

sudo roamswitch server config set forward_min_severity medium
sudo roamswitch server restart
sleep 5
: > /tmp/syslog.log
sudo sh -c 'echo "# lab comment" >> /root/.ssh/authorized_keys'
sleep 10
cat /tmp/syslog.log
sudo roamswitch forward status

<36>1 2026-11-12T01:55:09.267Z ubuntu roamswitch 1634 timeline.fim [roamswitch@32473 event_id="18e7296969615736-1634" rule="" action="notified" severity="medium" pid="1860" proc="dash"] /root/.ssh/authorized_keys: modified (login path not determined: proc)
Event forwarding (SIEM/syslog/webhook) - opt-in, default OFF
  enabled: yes
  running: submitted=1 dropped(oldest)=0 retry queue=0 items / 0 bytes
  - syslog_udp: ok=1 failed=0 last error=-
  - file: ok=1 failed=0 last error=-
  - webhook: ok=1 failed=0 last error=-

Only the timeline.fim event arrived. The low copy of the alert was not sent, and submitted is 1: events below the floor are not counted. The alert that was not forwarded still went into the notification history. When you are done, put the file back with the same command as above.

Put the floor back to low, then stop the webhook receiver. Find the listener's PID with ss and stop it.

sudo roamswitch server config set forward_min_severity low
sudo roamswitch server restart
sleep 5
P=$(sudo ss -ltnpH 'sport = :8099' | sed -n 's/.*pid=\([0-9]*\).*/\1/p'); kill $P
sudo roamswitch forward test; echo "exit=$?"
sudo roamswitch forward queue

  OK   syslog_udp
  OK   file
  FAIL webhook: HTTP 0
exit=2
retry queue: 0 (0 bytes) /var/lib/roamswitch/forward_queue.jsonl

HTTP 0 means no response came back. When any destination fails, forward test exits with code 2 (with no destination at all, 1). The test event does not go into the retry queue. The queue is for real events, so cause a real detection.

sudo sh -c 'echo "# lab comment" >> /root/.ssh/authorized_keys'
sleep 10
sudo roamswitch forward status
sudo roamswitch forward queue

Event forwarding (SIEM/syslog/webhook) - opt-in, default OFF
  enabled: yes
  running: submitted=2 dropped(oldest)=0 retry queue=2 items / 1040 bytes
  - syslog_udp: ok=2 failed=0 last error=-
  - file: ok=2 failed=0 last error=-
  - webhook: ok=0 failed=4 last error=HTTP 0
retry queue: 2 (1040 bytes) /var/lib/roamswitch/forward_queue.jsonl
  2026-11-12T01:55:31.151Z webhook  /root/.ssh/authorized_keys: modified (no SSH login found: service tart-guest-agent.service)
  2026-11-12T01:55:31.209Z webhook  Critical Path FIM Tampering Detected (1 violation): Host: ubuntu (IP: 192.168.64.24) / Integrity violations in critical system files: / /root/.ssh/authorized_keys: modified / Process: root / /usr/bin/dash / command: sh -c echo "# lab comment" >> /root/.ssh/authorized_keys (pid 2198) / No SSH login was found (it is a process of the systemd service tart-guest-agent.service) / Recommended Action: Run `sudo roamswitch fim verify` to inspect changes. / Timestamp: 2026-11-12 01:55:31

syslog and the file got both events. Only the two webhook events stayed in the queue. Start the receiver again.

: > /tmp/fwd.log
nohup python3 /tmp/recv.py >/dev/null 2>&1 &
sleep 20
sudo roamswitch forward queue
sudo roamswitch forward status

retry queue: 0 (0 bytes) /var/lib/roamswitch/forward_queue.jsonl
Event forwarding (SIEM/syslog/webhook) - opt-in, default OFF
  enabled: yes
  running: submitted=2 dropped(oldest)=0 retry queue=0 items / 0 bytes
  - syslog_udp: ok=2 failed=0 last error=-
  - file: ok=2 failed=0 last error=-
  - webhook: ok=2 failed=4 last error=-

Within 20 seconds the queue was empty. /tmp/fwd.log holds the two events from while it was down, with their original ts. Once the retry succeeded, last error went back to -. The failed count keeps the failures from while it was down. When you are done, put the file back.

sudo roamswitch notifications

=====================================================
  🔔 Notification History (past 7 days)
=====================================================

  2026-11-12T01:55:31.209488464+00:00
  Critical Path FIM Tampering Detected (1 violation)
    Host: ubuntu (IP: 192.168.64.24)
    Integrity violations in critical system files:
    /root/.ssh/authorized_keys: modified
      Process: root / /usr/bin/dash / command: sh -c echo "# lab comment" >> /root/.ssh/authorized_keys (pid 2198)
      No SSH login was found (it is a process of the systemd service tart-guest-agent.service)
    Recommended Action: Run `sudo roamswitch fim verify` to inspect changes.
    Timestamp: 2026-11-12 01:55:31

  Total: 4 notifications
  This is the record of the notifications RoamSwitch raised. It does not show whether they arrived (they are recorded even when no channel is set up). To check delivery: sudo roamswitch server test-notify

Read it like this.

• The leading time is UTC (+00:00). In Japan time, add nine hours: 10:55 on November 12
• It names the writing process, the command, and whether an SSH login was involved. When the process had already exited, a note says the details come from the exec record
• As the last two lines say, it does not show whether an alert arrived. Alerts are recorded even with no channel set up. Check delivery with test-notify

The text is shown in the reader's language. In this check, the alerts on this machine were recorded with language set to en, and reading them with LANG=ja_JP.UTF-8 showed them in Japanese. Addresses, paths and command lines stay as recorded. The forwarded copies and the text sent to alert channels use the language in effect when they were sent.

To hand the history to another tool, take it out as JSON or JSON Lines.

sudo roamswitch notifications --json | python3 -c 'import json,sys; print(sorted(json.load(sys.stdin)[0]))'
sudo roamswitch notifications --export /tmp/notif.jsonl
ls -l /tmp/notif.jsonl

['body', 'text', 'timestamp', 'title']
Exported 4 records to /tmp/notif.jsonl as JSON Lines.
-rw------- 1 admin admin 6542 Nov 12 01:56 /tmp/notif.jsonl

Each record has four fields: timestamp, title, body and text. title and body are text in the reader's language, and text is a language-neutral form (message keys and values). There is no severity field. The exported file is -rw------- and belongs to the user who ran sudo (admin here). The body includes command lines, so check the contents before you hand the file to anyone.