A dependency was hijacked | Learn Defense with RoamSwitch

#The situation

Right after a deploy, two notifications arrive

The first is "Dependency Lockfile Changed." The second is "A decoy credential file was opened," and what opened it was node setup.js.

A moment ago, you ran npm update on the production server to refresh its dependencies.

When a dependency is hijacked, a malicious version is published under the legitimate name. An ordinary update pulls it in. This module is a playbook for what an operator does after that. The mechanisms and the checks themselves are covered in Supply-chain attacks.

Here you take one scenario through five steps: stage it, read the detections, investigate, respond, and undo. The features used are Lock file tamper monitoring, Credential decoys, Matching packages against known vulnerabilities, Check of code that runs at install time, Process execution recording and attack-technique detection, Incident timeline, Notification history, and API key and private key detection.

The subject is the real debug incident. On 8 September 2025, the npm publishing account for debug was taken over through phishing. Version 4.4.2 was published, and the fix is 4.4.3. In the labs you build harmless dummies with that name and those versions.

#Stage the scenario

This changes the system

The lab changes two server settings and creates projects under /srv. Use a throwaway VM. The last chapter undoes everything.

Set up the dummy packages, a registry and the projects

You need npm and git (on Ubuntu, sudo apt-get install -y npm git). Build debug 4.4.1, 4.4.2 and 4.4.3. Each contains only a function that does nothing. Only 4.4.2 gets a script that runs at install time (postinstall). That script only opens and reads a decoy key file and sends nothing anywhere.

mkdir -p ~/lab/registry/debug/- && cd ~/lab
for v in 4.4.1 4.4.2 4.4.3; do
  mkdir -p src/$v
  echo 'module.exports = () => () => {};' > src/$v/index.js
  echo "{ \"name\": \"debug\", \"version\": \"$v\", \"main\": \"index.js\" }" > src/$v/package.json
done
# Only 4.4.2 gets a script that runs at install time
# It only opens and reads a decoy key file; it sends nothing anywhere
cat > src/4.4.2/package.json <<'J'
{ "name": "debug", "version": "4.4.2", "main": "index.js",
  "scripts": { "postinstall": "node setup.js" } }
J
echo "try { require('fs').readFileSync('/root/.aws/credentials'); } catch (e) {}" > src/4.4.2/setup.js
for v in 4.4.1 4.4.2 4.4.3; do (cd src/$v && npm pack --silent --pack-destination ~/lab/registry/debug/-); done

Next, run a registry that listens on 127.0.0.1 only. npm sees only the versions you pass to publish.py. Nothing is published to an outside registry.

cat > ~/lab/publish.py <<'P'
# Choose which versions the registry lists (inside this VM only; nothing is published outside)
import base64, hashlib, json, os, sys, tarfile
root = os.path.expanduser("~/lab/registry/debug")
doc = {"name": "debug", "versions": {}, "dist-tags": {"latest": sys.argv[-1]}}
for v in sys.argv[1:]:
    path = f"{root}/-/debug-{v}.tgz"
    data = open(path, "rb").read()
    with tarfile.open(path) as t:
        meta = json.load(t.extractfile("package/package.json"))
    if "postinstall" in meta.get("scripts", {}):
        meta["hasInstallScript"] = True
    meta["dist"] = {"tarball": f"http://127.0.0.1:4873/debug/-/debug-{v}.tgz",
        "integrity": "sha512-" + base64.b64encode(hashlib.sha512(data).digest()).decode()}
    doc["versions"][v] = meta
json.dump(doc, open(f"{root}/index.json", "w"))
print("published:", " ".join(sys.argv[1:]))
P
cat > ~/lab/registry.py <<'P'
# A read-only registry listening on 127.0.0.1 only
import http.server, os
ROOT = os.path.expanduser("~/lab/registry")
class H(http.server.SimpleHTTPRequestHandler):
    def translate_path(self, path):
        full = os.path.join(ROOT, path.split("?")[0].strip("/"))
        return os.path.join(full, "index.json") if os.path.isdir(full) else full
http.server.ThreadingHTTPServer(("127.0.0.1", 4873), H).serve_forever()
P
nohup python3 ~/lab/registry.py >/dev/null 2>&1 &
python3 ~/lab/publish.py 4.4.1

Then create the production app /srv/shop and a separate batch job /srv/batch. Both depend on "debug": "^4.4.1", which installs 4.4.1 for now. /srv/shop gets a .env with a dummy key, and its lockfile is committed to git.

sudo mkdir -p /srv/shop /srv/batch
for app in shop batch; do
  echo "{ \"name\": \"$app\", \"private\": true, \"dependencies\": { \"debug\": \"^4.4.1\" } }" | sudo tee /srv/$app/package.json >/dev/null
  echo 'registry=http://127.0.0.1:4873/' | sudo tee /srv/$app/.npmrc >/dev/null
  (cd /srv/$app && sudo npm install --no-audit --no-fund --silent)
done
echo 'AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE' | sudo tee /srv/shop/.env >/dev/null
cd /srv/shop && sudo git init -q && sudo git add package.json package-lock.json .npmrc
sudo git -c user.name=ops -c user.email=ops@example.com commit -qm "debug 4.4.1"

Now change two RoamSwitch settings. lockfile_watch_dirs lists the project folders whose lockfiles are watched. It is empty by default, so nothing is watched. package_cve_map_languages_enabled turns on receiving vulnerability data for npm and other dependency ecosystems. It is off by default. Both need a restart to take effect.

sudo roamswitch server config set lockfile_watch_dirs /srv/shop
sudo roamswitch server config set package_cve_map_languages_enabled true
sudo roamswitch server restart
sudo systemctl start roamswitch-update.service
cat /var/lib/roamswitch/package_cve_map_npm.VERSION

Excerpt of the output

  ✓ Server configuration updated and saved. (lockfile_watch_dirs = /srv/shop)
  ✓ Server configuration updated and saved. (package_cve_map_languages_enabled = true)
  ✓ roamswitch-server service successfully restarted.
2026-11-11

The last line is the date of the npm vulnerability data received. If the file does not exist, the data has not arrived yet. Until it arrives, dependency checking does not work.

#Read the notifications

A hijacked version is published, and an update pulls it in

List 4.4.2 in the registry. This is the moment the publishing account is taken over and a malicious version appears. Then run npm update as usual.

python3 ~/lab/publish.py 4.4.1 4.4.2
cd /srv/shop && sudo npm update --no-audit --no-fund
sleep 10
sudo roamswitch notifications | head -40

Excerpt of the output

  2026-11-12T01:43:28.597332685+00:00
  A decoy credential file was opened
    /root/.aws/credentials (aws_credentials) is a decoy, not a real file.
    Process: root / /usr/bin/node / command: node setup.js (pid 7931)
    Parent processes: dash ← node ← sudo ← bash ← tart-guest-agent ← systemd / Service: tart-guest-agent.service

  2026-11-12T01:43:26.159321391+00:00
  Dependency Lockfile Changed (1 file)
    Host: ubuntu (IP: 192.168.64.18)
    Changed lockfiles:
    /srv/shop/package-lock.json: modified
      Process: root / /usr/bin/node / command: npm update (pid 7913)
      No SSH login was found (it is a process of the systemd service tart-guest-agent.service)
    npm/yarn/pnpm appeared to be running at the same time — likely a deploy's own install/ci, but worth a glance if none was expected.
    Recommended Action: Run `sudo roamswitch lockfile-fim verify` to inspect the diff, then `lockfile-fim update` to re-baseline once confirmed legitimate.
    Timestamp: 2026-11-12 01:43:26

Below these come the server's own check notices, such as "A protection you switched on is not working". The history is shown in the reader's language, so this output is English even though the server's language is ja.

The two notifications come from separate features.

Notification	Feature	What it tells you
Dependency Lockfile Changed	Lock file tamper monitoring	`/srv/shop/package-lock.json` differs from the recorded content. `npm update` wrote it
A decoy credential file was opened	Credential decoys	`node setup.js` opened the `/root/.aws/credentials` decoy. Its parent is `node`, whose parent is `sudo`

A lockfile change alone also happens in an ordinary update. The notification itself adds that npm/yarn/pnpm was running, so a deploy is the likely cause. A decoy being opened does not. A real app has no reason to read a decoy key. When both arrive together, investigate on the assumption that a hijacked version got in.

Decoys are placed under honeytokens_home (/root by default). In the lab VM the commands go through tart-guest-agent, so it appears among the parents. That part differs when you log in over SSH.

#When and where it got in

Find the affected version, when it arrived and what ran

First, check the lockfiles against the known-vulnerability data. List several folders to check them at once.

sudo roamswitch scan-packages /srv/shop /srv/batch | sed -n '/npm\/PyPI/,$p'

Excerpt of the output

  Needs action: 1 / accepted: 0 (of which ending within 14 days: 0) / acceptance ran out and counted open again: 0

  CVE-2025-59144  [npm] debug 4.4.2 (CVSS 7.0)
    EPSS 0.5% (2026-11-10)  (an estimate of how likely exploitation is, not a confirmation that it is exploited)
    📁 /srv/shop/package-lock.json
    Fixed in: 4.4.3
    Impact On 8 September 2025, the npm publishing account for debug was taken over after a phishing attack. Version 4.4.2 was published, functionally identical to the previous patch version, but with a malware payload added attempting to redir…

Only /srv/shop matches; /srv/batch does not appear. Matching packages against known vulnerabilities reads the versions written in the lockfile and runs nothing.

Next, look across the whole host for where that version sits. Also check what changed and when it arrived.

sudo find / -xdev -path '*/node_modules/debug/package.json' -exec grep -H '"version"' {} +
cd /srv/shop && sudo git diff package-lock.json | grep '^[-+] *"version"'
stat -c '%y %n' /srv/shop/node_modules/debug/package.json
sudo sh -c 'grep -H "argv" /root/.npm/_logs/*.log'

Output

/srv/batch/node_modules/debug/package.json:{ "name": "debug", "version": "4.4.1", "main": "index.js" }
/srv/shop/node_modules/debug/package.json:{ "name": "debug", "version": "4.4.2", "main": "index.js",
-      "version": "4.4.1",
+      "version": "4.4.2",
2026-11-12 01:43:26.008933274 +0000 /srv/shop/node_modules/debug/package.json
/root/.npm/_logs/2026-11-12T01_42_51_869Z-debug-0.log:21 verbose argv "install" "--no-audit" "--no-fund" "--loglevel" "silent"
/root/.npm/_logs/2026-11-12T01_42_52_480Z-debug-0.log:21 verbose argv "install" "--no-audit" "--no-fund" "--loglevel" "silent"
/root/.npm/_logs/2026-11-12T01_43_25_934Z-debug-0.log:21 verbose argv "update" "--no-audit" "--no-fund"

4.4.2 exists only in /srv/shop. The git diff shows the move from 4.4.1 to 4.4.2, and the file time and npm's log show it came from the npm update at 01:43 (UTC). The npm installed with Ubuntu's apt keeps its own debug in /usr/share/nodejs/debug (4.3.4 on this VM). That is not under node_modules, so this find does not show it.

Last, list what the new version runs at install time. Then confirm from the exec record that it actually ran. Replace the PID in events tree with your own.

sudo roamswitch scan-scripts /srv/shop | sed -n '/an inventory/,$p'
sudo roamswitch events search --since 1h --text setup.js | grep -v 'events search'
sudo roamswitch events tree 7931

Excerpt of the output

  This is an inventory, not a threat verdict — nothing is ever executed. ⚠️ is a lightweight heuristic that includes false positives.

    debug 4.4.2 [postinstall]
    node setup.js

  No state has been marked as reviewed yet. After checking the current install scripts, run roamswitch scan-scripts --accept. From then on, only what has been added is reported.

2026-11-12T01:43:26.015Z pid=7931 ppid=7930 root /usr/bin/node setup.js
2026-11-12T01:43:26.015Z pid=7930 ppid=7913 root /usr/bin/dash -c node setup.js
1 /usr/lib/systemd/systemd
694 /usr/bin/tart-guest-agent
7842 /usr/bin/bash
7912 /usr/bin/sudo
7913 /usr/bin/node
7930 /usr/bin/dash
              7931 /usr/bin/node setup.js

Check of code that runs at install time shows that 4.4.2 has a postinstall. The Process execution recording and attack-technique detection record shows node setup.js running as root. Its parent, 7913, is npm update (sudo roamswitch events search --pid 7913 shows the arguments; an empty or non-numeric --pid exits with code 2 instead of listing everything).

#Estimate whether keys were stolen

Install-time scripts run with the privileges of whoever ran npm. Here that was sudo npm update, so root.

Check the timeline and the keys on disk

Use Incident timeline to see the detections in one timeline. Then use API key and private key detection to find keys stored in the project. The keys it finds are candidates for rotation.

sudo roamswitch timeline
sudo roamswitch audit-secrets /srv/shop

Excerpt of the output

  2026-11-12T01:43:26.108053902+00:00  [Honeytoken]  open
    ハニートークン(/root/.aws/credentials)への実アクセスを受け、関連する新規デコイを追加配置しました: /root/.aws/config
    MITRE ATT&CK: T1552 (Unsecured Credentials)
    action taken: decoy access recorded

  2026-11-12T01:43:26.078560448+00:00  [FIM]  recorded only
    /srv/shop/package-lock.json: modified (no SSH login found: service tart-guest-agent.service)
    process: node (pid 7913)
    action taken: notified

  2026-11-12T01:43:26.074832374+00:00  [Honeytoken]  open
    ハニートークン(aws_credentials)へのアクセスを検知: /root/.aws/credentials — アクセス元: プロセス: root / /usr/bin/node / コマンド: node setup.js (pid 7931) / 親プロセス: dash ← node ← sudo ← bash ← tart-guest-agent ← systemd / サービス: tart-guest-agent.service
    process: node (pid 7931)
    process ancestry: dash ← node ← sudo ← bash ← tart-guest-agent ← systemd
    MITRE ATT&CK: T1552 (Unsecured Credentials)
    action taken: decoy access recorded
(…)
  /srv/shop/.env
    ⚠️  AWS Access Key ID (AWS) — line 1
        Masked: AKIAIO...MPLE
        ↳ Recommendation: Deactivate the access key in the AWS IAM console.

  Found 1 exposure across 1 file. Address these before committing or sharing.

In the timeline, the lockfile change and the opened decoy fall in the same second (01:43:26), newest first. The decoy line names the command that opened it and its ancestry, with the same names as the notification. The text of the two Honeytoken lines stays in the server's language at the time they were recorded (language in server.conf; ja on this VM), so it is shown here as it appeared.

After the decoy was read, RoamSwitch placed one more decoy at /root/.aws/config. audit-secrets masks the .env key, showing only its start and end.

It found a key, so the exit code is 1 (0 when it finds nothing).

Decide which keys to rotate

Did the script run? If not, installing could not have read a key. Here the exec record has node setup.js
With what privileges? Treat everything those privileges can read as read. Here it was root, so that includes /root, /srv/shop/.env and the settings under /etc
Was a decoy read? If so, that is strong evidence of code that hunts for keys. If not, that does not mean nothing was stolen

If the answer to 1 and 2 is yes, rotate every real key those privileges can read. Issue a new key, put it in the app, confirm the app works, then disable the old key. Look for use of the old key in the records of whoever issued it (a cloud console, for example).

Check yourself

No decoy notification appeared. The install-time script ran as root. What can you say?

Show the explanation

A decoy is a way to catch code that hunts for keys. Code that reads only the real keys does not trip it. Decide by whether the script ran and with what privileges.

#Pin the version and reinstall

Pin the fixed version and reinstall without scripts

Remove 4.4.2 from the registry and list 4.4.3. This is the moment the hijacked version is pulled and a fix is published.

--save-exact writes the version into package.json without ^. With ^4.4.1 left in place, the next npm update may pull a different version that matches the range. --ignore-scripts is npm's switch for not running install-time scripts. Delete node_modules and reinstall exactly from the lockfile with npm ci.

python3 ~/lab/publish.py 4.4.1 4.4.3
cd /srv/shop
sudo rm -rf node_modules
sudo npm install debug@4.4.3 --save-exact --ignore-scripts --no-audit --no-fund
grep '"debug"' package.json
sudo rm -rf node_modules && sudo npm ci --ignore-scripts --no-audit --no-fund
sudo roamswitch scan-packages /srv/shop | sed -n '/npm\/PyPI/,$p'
sudo roamswitch scan-scripts /srv/shop | grep -F 'No install scripts'
sudo git -c user.name=ops -c user.email=ops@example.com commit -qam "debug を 4.4.3 に固定" && sudo git log --oneline

Excerpt of the output

published: 4.4.1 4.4.3
{"name":"shop","private":true,"dependencies":{"debug":"4.4.3"}}
  None of the 1 dependency that could be checked matched a known high-severity CVE (CVSS >= 7.0).
  No install scripts found.
8588234 debug を 4.4.3 に固定
cfa7a5e debug 4.4.1

The CVE match is gone, and so is the install-time script. Commit the fixed lockfile to git.

Because the lockfile was rewritten, the watch sends another notification.

Excerpt of the notification (sudo roamswitch notifications)

  Dependency Lockfile Changed (1 file)
    Host: ubuntu (IP: 192.168.64.18)
    Changed lockfiles:
    /srv/shop/package-lock.json: modified
      Process: root / /usr/bin/node / command: npm install debug@4.4.3 (pid 8484)

The writing command matches the npm install debug@4.4.3 you ran, and the git history agrees. Confirm from the notification itself that the change is yours before moving on.

scan-scripts also points out that .npmrc has no min-release-age (a setting that holds back freshly published versions). Check that your npm supports it before adding it.

Do not

Delete node_modules as soon as the notification arrives. Deleting before you investigate loses which version was there and the file times
Stop after installing the fixed version. If the script ran, rotating the keys is still left
Keep running npm update in production with a ^ range. The next hijack comes in the same way
Install dependencies as root on a production server. The scripts then run as root, with the widest possible reach

#Undo

Restore the settings and remove the lab files

Put the settings back to their defaults and restart, stop the registry and delete the lab files. lockfile_fim.db and lockfile_fim_notified.txt are the lockfile watch's records.

sudo roamswitch server config set lockfile_watch_dirs ''
sudo roamswitch server config set package_cve_map_languages_enabled false
sudo roamswitch server restart
kill $(sudo ss -ltnpH 'sport = :4873' | sed -n 's/.*pid=\([0-9]*\).*/\1/p')
sudo rm -rf /srv/shop /srv/batch ~/lab /root/.npm
sudo rm -f /var/lib/roamswitch/lockfile_fim.db /var/lib/roamswitch/lockfile_fim_notified.txt
sudo grep -E '^(lockfile_watch_dirs|package_cve_map_languages_enabled)=' /etc/roamswitch/server.conf
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4873/debug

Excerpt of the output

  ✓ Server configuration updated and saved. (lockfile_watch_dirs = )
  ✓ Server configuration updated and saved. (package_cve_map_languages_enabled = false)
  ✓ roamswitch-server service successfully restarted.
lockfile_watch_dirs=
package_cve_map_languages_enabled=false
000

The final 000 means the registry is stopped and the connection failed. The npm vulnerability data already received (/var/lib/roamswitch/package_cve_map_npm.json and others) stays after you turn the setting off. The /root/.aws/config decoy was placed by RoamSwitch, so leave it.

#What this cannot show

What this procedure cannot tell you

What the script did is known only as far as the exec record and the decoys reach. Reading a real key file does not show up as a decoy notification
Sending data out was not staged in this lab. Whether anything was sent has to be checked in other records (outbound connection logs, for example)
The lockfile watch covers only the folders in lockfile_watch_dirs. The lab's /srv/batch was not registered, so it was not watched
scan-packages matches only versions in the vulnerability data received. A hijack that is not in the data yet does not show up
scan-scripts lists install-time scripts. Code that runs when a package is required is not listed

Check yourself

After pinning the fixed version and reinstalling, what work is still left?

Show the explanation

Fixing the version stops what comes in from now on. Whatever the script that already ran may have read is not undone by fixing the version.

What to learn next

Install-time code checks and look-alike package names are covered in Supply-chain attacks
For how to think when a key may have leaked, see Credential theft and leaked secrets
The first steps after containment kicks in are in A containment fired: the first hour

Features covered

Related techniques (ATT&CK): T1195.001 T1552.001

What was verified

Verified on 2026-11-12. Target: Linux 1.17.39. On a disposable Ubuntu 24.04 VM (published 1.17.39 Server Edition, with apt's npm and git), the lesson's steps were run from scratch using a registry bound to 127.0.0.1 and harmless dummy debug 4.4.1 to 4.4.3. English output was taken on the same VM by switching LANG, with the server's language left at ja.

Command output varies with the product version and the environment. If yours differs, check the version first.