sdlc.dev

Node.js path · Doing more at once · Topic 7 of 8

Streams and backpressure

Not started

A stream moves data in chunks instead of all at once. Readable streams produce chunks, Writable streams consume them, Duplex streams do both and Transform streams change chunks on the way through. Each stream buffers up to its highWaterMark; when a writable's buffer is full, write() returns false, and a well-behaved producer waits for the 'drain' event before writing more. That signal is backpressure.

Why it matters: Without backpressure a fast producer fills memory until the process runs out; with it, a gigabyte file passes through a few kilobytes of buffer.

1

How it works

The explanation, with examples you can run.

Let pipeline do the work

// gzip.mjs (run with: node gzip.mjs big.log)import { createReadStream, createWriteStream } from "node:fs"import { pipeline } from "node:stream/promises"import { createGzip } from "node:zlib" const file = process.argv[2]await pipeline(createReadStream(file), createGzip(), createWriteStream(`${file}.gz`))console.log("done")

pipeline reads only as fast as gzip and the disk can take, passes any error on, and closes every stream if one fails. Reading with for await (const chunk of readable) also respects backpressure: the next chunk is pulled only when your loop asks for it.

Writing by hand

If you call write() yourself, check what it returns.

import { once } from "node:events" async function writeLines(stream, lines) {  for (const line of lines) {    if (!stream.write(`${line}\n`)) await once(stream, "drain")  }  stream.end()}

Ignoring false still works, but every extra chunk piles up in memory. The limit is highWaterMark: a byte count for byte streams (64 KiB by default in Node 22; stream.getDefaultHighWaterMark() tells you) or an object count in object mode.

The same idea in the browser

Web Streams (ReadableStream, also global in Node) work on a pull model: the source is asked for more only while the queue has room. Here the reader is slow, so the source stops after filling the queue to its high water mark of 2.

⌘/Ctrl + Enter
OutputNot run yet

Press Run to see what your code prints, in order.

Runs in this browser, in an isolated frame with no access to this page. console.log, info, warn and error are shown; code runs as a classic script, so use an async function for await.

2

Check yourself

2 questions. Get them all right to finish this topic.

  1. 1.stream.write(chunk) returned false. What should the producer do?
  2. 2.Why prefer stream.pipeline() over a.pipe(b).pipe(c)?

0 of 2 right so far. Get all two to finish this topic.

Go deeper