event-stream compromise

In October 2018 I investigated why nodemon was emitting a strange deprecation warning, leading to the eventual discovery of the now infamous event-stream npm package compromise. This post shares a unique perspective of the events that unfolded, along with downloadable forensic evidence that has since been purged from the npm registry.

For context, see:

• dominictarr/event-stream#116
• npm – “Details about the event-stream incident”
• Snyk – “A Post-Mortem of the Malicious event-stream backdoor”

In the media:

• ZDNet – “Hacker backdoors popular JavaScript library to steal Bitcoin funds”
• Ars Technica – “Widely used open source software contained bitcoin-stealing backdoor”
• The Register – “Check your repos... Crypto-coin-stealing code sneaks into fairly popular NPM lib (2m downloads per week)”

Dates and times mentioned or pictured in this post are Australian Eastern Standard Time (AEST).

Forensic evidence

As the compromised and malicious releases have been unpublished from the npm registry and purged from most mirrors and CDNs, for security research purposes you may download the archive event-stream-compromise-evidence.zip:

event-stream-compromise-evidence
├── event-stream@3.3.6
│   ├── LICENCE
│   ├── examples
│   │   ├── data
│   │   ├── map.js
│   │   ├── pretty.js
│   │   └── split.js
│   ├── index.js
│   ├── package.json
│   ├── readme.markdown
│   └── test
│       ├── connect.asynct.js
│       ├── flatmap.asynct.js
│       ├── helper
│       │   └── index.js
│       ├── merge.asynct.js
│       ├── parse.asynct.js
│       ├── pause.asynct.js
│       ├── pipeline.asynct.js
│       ├── readArray.asynct.js
│       ├── readable.asynct.js
│       ├── replace.asynct.js
│       ├── simple-map.asynct.js
│       ├── spec.asynct.js
│       ├── split.asynct.js
│       ├── stringify.js
│       └── writeArray.asynct.js
├── flatmap-stream@0.1.1
│   ├── LICENSE.txt
│   ├── README.md
│   ├── index.js
│   ├── index.min.js
│   ├── package.json
│   ├── perf-test.js
│   └── test
│       └── data.js
└── flatmap-stream@0.1.2
    ├── LICENSE.txt
    ├── README.md
    ├── index.js
    ├── index.min.js
    ├── package.json
    ├── perf-test.js
    └── test
        └── data.js

Chances are you can source these unpublished package versions yourself by Finder searching the node_modules of a project npm installed in last month or so (but not yet updated) for flatmap-stream.

flatmap-stream@0.1.1 contains the malicious code and is the most interesting to look at.

Delayed response

I first caught wind something was fishy a month ago (October 29, 2018), when I could't work out why nodemon was emitting a crypto deprecation warning in Node.js v11 (remy/nodemon#1442):

[DEP0106] DeprecationWarning: crypto.createDecipher is deprecated.

It was not until November 21, 2018 that Ayrton Sparling figured out it was due to an attack:

At which point I put out an urgent tweet:

It got very little attention. The JS community that day was apparently more interested in memes and React hooks shitposts:

It took far too long for the community to mobilize.

I attempted to report the threat to npm the next day, as the event-stream maintainer didn't seem so inclined (a reliable instinct). At first it was not obvious the right way to go about it; I filed a report at npmjs.com/advisories/report, but after submitting it redirected to a 404 page, I got no confirmation email or anything, and there appeared to be no place to track the status. So I tweeted npm to make sure they got it:

npm didn’t directly reply; they only took down the malicious packages after the story went viral a week later. Their tweet about the incident (November 28, 2018) claimed they were notified “yesterday morning”:

I replied that in reality, npm was notified a week earlier:

40 minutes later npm Marketing Vice President @cowperthwait initiated a direct message conversation, asking about what went wrong. He revealed that npm was not monitoring Twitter mentions over the US Thanksgiving holiday weekend. Aparently no one checked the mentions upon returning to work either.

npm then put out a new tweet, dropping the wording about when they were notified:

The nodemon maintainer first responded 22 hours after the threat was detected:

A nodemon version without compromised dependencies was publish relatively soon after. nodemon had a head start; the majority of the JavaScript community continued blissfully unaware for a week before the story went viral.

Reflections

Code on GitHub & npm can differ

It is problematic that package source code on Github, what most people scrutinize, can be totally different to what is published to npm. Hackers are exploiting this to undermine the open source “many eyes” security advantage.

For packages with a repository field, the npm publish command could accept a commit hash argument and the actual packaging could take place on npm servers, pulling the relevant commit. This will have the nice side-effect of ensuring packages can be built reliably from dev dependencies and scripts in environments other than the original author’s.

Security report gatekeeping

• The reporting form could be more obvious and better tested.
• Reports via alternative channels such as Twitter should not be ignored.
• There is always a delay between the initial community report and an official advisory or response, if there is one.

It would be nice if there was a way to audit your projects for unconfirmed community reports of active security threats. There needs to be a clear distinction in the reporting process between security vulnerabilities, which should not be immediately disclosed to the public, and active threats, which should.

Erased history

Instead of removing bad versions of compromised or malicious packages from the npm registry as if they never existed, they should be made only installable with a special flag. This would reduce confusion and allow security researchers to do their thing without having to contact each other asking for copies.