Crisis response for server operators

A web shell was found

Read the web shell and process alerts, trace the entry through the exec record and access log, list leaked keys, and take the vaulted file out of the public folders to examine it.

  • Level: Practice
  • About 40 min
  • 2026-11-12

#The situation

Your monitoring shows a "web shell detected" alert

The site you run is nginx with php-fpm. The alert reads "The web application rules detected a web shell or injected code", and names one PHP file under the public uploads/ folder. A little earlier, another alert said "A shell was spawned by a web/app service process". Nothing was blocked (both are record-and-notify only).

This playbook is about what to do when a web shell is found: how to read the detections, investigate, and restore. It does not cover attack techniques. File detection uses Web shell and malicious code detection, process detection uses Process execution recording and attack-technique detection, the timeline uses Incident timeline, the access-log investigation uses Access log diagnosis (AccessScope), and the check for leaked keys uses API key and private key detection.

The order is the same as other playbooks: confirm, investigate, respond, recover. Before you delete anything or take the site down, pin down what happened, on a timeline.

#Read the file detection

RoamSwitch scans a web file the moment it is closed after being written under a watched document root (those in the nginx and other server configs, plus folders you add). A file placed by a move such as mv is scanned right after it lands, too. On a match, by default it moves only critical findings to the quarantine vault and reports the rest in place.

Trigger a detection and read the alert

To confirm the detection, place a single file whose contents are the same harmless string the detection rule's unit test uses. It does nothing harmful. After placing it, wait a few seconds for the write watcher to pick it up.

# detection check only; contents match the rule's test string
printf '<?php\n@eval($_POST["x"]);\n' | sudo -u www-data tee /var/www/html/uploads/widget.php
sleep 6
roamswitch notifications      # notification history for the past 7 days
excerpt
  2026-11-12T02:56:13.326243587+00:00
  Web rules: 1 file matched
    🚨 The web application rules detected a web shell or injected code
    Host: ubuntu (IP: 192.168.64.6)
    Matched files: 1 (quarantined: 1, left in place: 0)
    /var/www/html/uploads/widget.php (php-request-exec, quarantined, owner www-data)
    Recommended Action: review the files and the access log around the time they were written, and change the credentials of that site (admin panel, FTP, database). Quarantined files are listed in `/var/lib/roamswitch/quarantine/metadata.json`.
    Timestamp: 2026-11-12 02:56:13
    Process: www-data / /usr/bin/tee / command: tee /var/www/html/uploads/widget.php (pid 7681) (The process had already exited, so this comes from the exec record made at 2026-11-12 02:56:13.)
    Parent processes: sudo ← dash ← tart-guest-agent ← systemd / Service: tart-guest-agent.service

Read the alert like this:

  • The matched-files line names the file, the rule it hit, and whether it was quarantined or left in place. php-request-exec is the rule name, quarantined means it was moved, and owner www-data is the owner
  • The recommended action is to review the contents and the access log, and change the site's credentials. The vault path for the quarantined file is in metadata.json
  • Process and parent processes are the thing that wrote the file: the account, the executable and the command line. Here tee had already exited before the alert, so the fields come from the exec record, and the line ends with "(The process had already exited, so this comes from the exec record made at …)". If the writer is still running, that note is not added
  • Timestamps are UTC. They are 9 hours ahead of Japan time. Keep that in mind when you line them up with the access log

#Read the process detection

Separately from the file detection, the exec record detects a web server process spawning a shell. That is EXR-001. It corresponds to a web shell running an external command. It does not block; it only notifies.

Make a service process spawn a shell

Place a PHP file that only runs the harmless id command, then open it once from localhost. This reproduces php-fpm spawning a shell as a child.

printf '<?php system("id");\n' | sudo -u www-data tee /var/www/html/uploads/probe.php
sleep 6
curl -s -o /dev/null http://127.0.0.1/uploads/probe.php
sleep 5
roamswitch notifications      # EXR-001 is at the top
roamswitch timeline           # unified incident timeline
excerpt
  2026-11-12T02:56:25.358320959+00:00
  🔎 Suspicious execution pattern detected
    A shell was spawned by a web/app service process (EXR-001)
    Process: /usr/bin/dash (PID: 7762) / triage: High
    Details: dash spawned by a service process as user www-data; command: sh -c -- id; parent service: /usr/sbin/php-fpm8.3 (pid 7196)
    Next step: sudo roamswitch events tree 7762
    (notification only — nothing was blocked)
    Process: www-data / /usr/bin/dash / command: sh -c -- id (pid 7762)
    Parent processes: php-fpm8.3 ← php-fpm8.3 ← systemd

  2026-11-12T02:56:19.376993944+00:00
  Web rules: 1 file matched
    🚨 The web application rules detected a web shell or injected code
    Host: ubuntu (IP: 192.168.64.6)
    Matched files: 1 (quarantined: 0, left in place: 1)
    /var/www/html/uploads/probe.php (php-code-in-uploads, reported, owner www-data)

  🕒 Unified Incident Timeline

  2026-11-12T02:56:25.379220246+00:00  [Exec Recorder]  recorded only
    EXR-001: dash spawned by a service process as user www-data; command: sh -c -- id; parent service: /usr/sbin/php-fpm8.3 (pid 7196)
    process: /usr/bin/dash (pid 7762)
    process ancestry: php-fpm8.3 ← php-fpm8.3 ← systemd
    MITRE ATT&CK: T1505.003 (Web Shell)
    action taken: notify

Three things to take from this:

  • The parent service is php-fpm8.3. The web server's run user (www-data) spawned a shell from the web server process. Ordinary web requests do not do that
  • The timeline shows T1505.003 (Web Shell). The file detection and the process detection are two faces of the same event
  • The alert raised when probe.php was placed is the php-code-in-uploads rule: it calls an execution function such as system in a folder that should only hold data. It is below critical, so it is reported and left in place. This was the only alert for that file (no second one came within a minute)
The file and process detections appear separately

The file detection fires when the file is placed; the process detection fires when the file is opened and spawns a shell. Sometimes you only get one. Put them together and you get both the time it was placed and the time it was run.

The same hit is notified once in 5 minutes. If the file is opened again within that time, only the timeline gets a row, marked "(recorded only: the same hit is notified once in 5 minutes, and at most 30 notifications a minute, so this one was not)".

#Investigate how it got in

Next, work out who did what, and when. These are read-only commands; they do not change files.

Line up the exec record against the access log

Look at the exec record for the run user www-data and at the access log. Line up the POST times in the access log with the file modification times to see how the file arrived.

sudo roamswitch events search --since 30m --user www-data    # www-data's exec record
sudo accessscope scan /var/log/nginx/access.log              # access-log diagnosis
stat -c '%n %U %y' /var/www/html/uploads/*                   # owner and mtime
excerpt
2026-11-12T02:56:25.337Z pid=7763 ppid=7762 www-data /usr/bin/id
2026-11-12T02:56:25.336Z pid=7762 ppid=7196 www-data /usr/bin/dash -c -- id
2026-11-12T02:56:19.322Z pid=7718 ppid=7717 www-data /usr/bin/tee /var/www/html/uploads/probe.php
2026-11-12T02:56:13.223Z pid=7681 ppid=7680 www-data /usr/bin/tee /var/www/html/uploads/widget.php

## Check first (possible successes)

| # | Severity | Confidence | Finding | Count | Main sources |
|---|---|---|---|---|---|
| 1 | Medium | Hypothesis | Possible web shell access ※1 | 1 (1 IP) | 127.0.0.1(1) |

/var/www/html/uploads/probe.php www-data 2026-11-12 02:56:19.320989293 +0000
  • events search lists the commands www-data ran, with timestamps. The exec record is readable by root only, so use sudo. It confirms again that the parent of dash -c -- id is php-fpm (pid 7196)
  • accessscope scan reads the access log and raises "Possible web shell access" with its evidence. "Hypothesis" confidence means a suspicion only: a response came back, but with no command parameter and no use right after
  • Line up the mtime from stat with the POST time in the access log to tell which request placed the file
Lining up the timestamps

Access-log times are in the server's time zone; the RoamSwitch alerts and events search are in UTC. Convert to one or the other before you compare them.

#Leaked secrets, and restoring from quarantine

Once a web shell is placed, the contents of config files may have been read through it. Check what keys and passwords the site holds, and rotate them.

List the secrets in the config files

Scan the public folder with audit-secrets. It only reads; it does not change files.

sudo roamswitch audit-secrets /var/www/html; echo "exit code: $?"
excerpt
  Scanned 3 files.

  /var/www/html/config.php
    ⚠️  AWS Access Key ID (AWS) — line 3
        Masked: AKIAIO...MPLE
        ↳ Recommendation: Deactivate the access key in the AWS IAM console.

  Found 1 exposure across 1 file. Address these before committing or sharing.
exit code: 1

The exit code is 1 when something is found and 0 when nothing is. A path that does not exist gives 2, so a monitoring script can tell the cases apart.

Here config.php holds an AWS access key ID (this course uses the published, invalid sample value AKIAIOSFODNN7EXAMPLE). Treat any key found as leaked: deactivate it from the issuer's console and replace it with a new one. Rotating keys and passwords is work you do outside RoamSwitch.

Check the vault, and restore if needed

Check where the quarantined file went. To examine its contents again, take it out to a place outside the public folders (this changes state).

sudo roamswitch quarantine list                  # vault contents (readable by root only)
sudo roamswitch web-scan /var/www/html            # dry run (no network)
sudo install -d -m 700 /srv/inspect               # a place to examine it, outside the public folders
sudo roamswitch quarantine restore quar-20261112_025613-33eef100 /srv/inspect/widget.php
sudo ls -l /srv/inspect
excerpt
  Vault location: /var/lib/roamswitch/quarantine
  Quarantined: 1 file

  • [quar-20261112_025613-33eef100] widget.php (WebShell-PHP: php-request-exec) - 26 bytes
    Original path: /var/www/html/uploads/widget.php

  Rules 2026-11-11 (20 rules)   Scanned 3 web files

  [HIGH] /var/www/html/uploads/probe.php
      php-code-in-uploads (WebShell-PHP)
      A PHP file in an upload/media directory calls an execution or decoder function. These directories should only hold user data.
      → not touched (dry run; add --quarantine to move files)

✓ Restored quar-20261112_025613-33eef100 (without the execute bit).
total 4
-rw-r--r-- 1 root root 26 Nov 12 02:57 widget.php
  • quarantine list shows each vaulted file's ID, threat name, and original path. The vault path and SHA-256 are in /var/lib/roamswitch/quarantine/metadata.json. Without sudo it prints "The quarantine vault could not be read" and exits with 1 (it does not claim zero items)
  • web-scan is a dry run by default and does not move files (it quarantines only with --quarantine). The remaining probe.php shows up as HIGH
  • quarantine restore <ID> <path> puts the vaulted file at the given path and takes it out of the vault. It never sets the execute bit and never overwrites a file of the same name. It refuses system locations such as /root
  • Without a path, it restores to the original location. If that is a public folder, the file is scanned again right away, and a real web shell is quarantined again at once
Quarantine, or leave it to observe

A file that hits a critical rule (such as php-request-exec) is quarantined by default, and stops being reachable at once. A below-critical finding like php-code-in-uploads is only reported, and left in place. Leaving it helps when you want to examine the state as-is. If you conclude it is a real intrusion, move the remaining files out too, close the entry path, and then recover. Restore from quarantine only when you have confirmed a false positive.

These steps change state

quarantine restore and rotating keys change state. Before you take a file out, keep metadata.json and the access log. After you confirm a false positive and put the file back at its original path, check with ls -l <original path> that the owner is the original account again (www-data here). When you finish examining a file you took out, delete it with sudo rm -r /srv/inspect.

#What this does not cover, and a check

Not verified in this course
  • The detection check used the same harmless string as the rule's unit test and a PHP file that only runs id. No real malware, and no web shell that takes commands from the outside, was used
  • Detection covers files written under a watched folder whose contents hit a rule. Something that runs code inside the web server process without writing a file is only seen, via EXR-001, when it spawns a shell from a service process. Obfuscation that slips past the rules, or writes outside the watched folders, is not seen
  • A file is read whole up to 64 MiB; larger ones are read at the head and tail only. Code appended to the end of an image is seen, but code in the middle of a file larger than 64 MiB is not
  • Rotating keys and passwords, fixing the entry path, and contacting the parties involved are done outside RoamSwitch and are out of scope here
Check yourself

A PHP file under uploads/ hit php-request-exec and the alert said quarantined. Where is the file now

Show the explanation

Hitting a critical rule means it was moved to the vault by default. Its original path and vault path are in sudo roamswitch quarantine list and metadata.json. It is not deleted, so a false positive can be put back with sudo roamswitch quarantine restore <ID>.

Next, the "Web and email" module in the attack track covers how web shell detection works, and the defenses that stop probing seen in the access log.

Features covered

Related techniques (ATT&CK): T1505.003 T1059.004 T1027 T1105 T1553.001 T1204.004

What was verified

Verified on 2026-11-12. Target: Linux 1.17.39. On disposable Ubuntu 24.04 VMs with the published 1.17.39, nginx and php-fpm, every lab was run in order in Japanese and English on separate VMs, using the rule's harmless test string and a PHP file that only runs id, and a one-minute wait confirmed that no second report came for the same file.

Command output varies with the product version and the environment. If yours differs, check the version first.