Node Streams: an overview

3 minute read Published:

What is a stream? Why would you use one?

Whether you know it or not, if you have coded anything in Node.js it is likely that you have used streams. An express server serves streams as responses and accepts streams as requests.

You can think of a stream as a path for chunked data to travel down. Data is buffered and sent along in chunks, like trucks down a highway. In this way you can send a large amount of data in a manageable way to different processes without sacrificing performance. The data passed through a stream can be read, written, and transformed.

Streams improve efficiency and memory usage when working with larger data sets. They allow you to read, write, and transform data on the fly without having to store large amounts of data in memory.

Imagine you are trying to send a large amount of medicine to an NGO who is in desperate need of spreading the medicine to people as soon as possible. Instead of sending all of it all at once, you could save countless lives by sending the medicine as soon as possible in smaller chunks. Doing this would reduce storage costs, and increase the speed at which people would receive the medicine. Using streams, in the same way, reduces memory impact and increases the speed at which you handle and send data.

There are three main types of streams and they are most likely piped together in this way:

readable.pipe(duplex).pipe(writable)

Example Usages

  • To read a file until a specified value that isn’t at the end.
  • To transform data on the fly.
  • To create a communication channel between two streams (websockets).
  • To send large files across the net (including the client)

How Node Internals Use Streams

  • The HTTP package uses streams for sending and receiving requests.
  • The Crypto package implements a transform stream for the Cypher class to create encrypted and decrypted data.
  • The File System package implements readable and writable streams to read and write files.
  • The Process package uses duplex streams for standard in, out, and error that allows you to send data to the application and get information out
  • The Net package uses duplex streams to create sockets. process.stdin and others implement net.Socket themselves.
  • The ZLib package uses transform streams to create compressed outputs of a given input.

Three Types of Streams

  1. Readable Streams (ex: http.IncomingMessage)
  2. Writable Stream (ex: http.ServerResponse)
  3. Duplex Stream (ex: net.socket)
    a. Transform Stream (ex: crypto.createCipher)

Let’s get into each one in detail using a number of examples. The full repo for these examples are available here, and were written for a talk I gave at an NYC JavaScript Meetup. For all of the following examples we’ll use this text file named tyrion.txt containing the following:

It’s not easy being drunk all the time. If it were easy, everyone would do it.
comments powered by Disqus