Streams and Buffers in Node.js – A Complete Guide

Streams and Buffers are fundamental concepts in Node.js that enable efficient handling of large volumes of data. Whether you’re reading files, sending data over a network, or piping output to another service, understanding streams and buffers is essential for writing high-performance, memory-efficient Node.js applications.

In this module, we’ll explore what Streams and Buffers are, how they work, their different types, and how to use them effectively in real-world applications.

Table of Contents

1. What Are Streams in Node.js?
2. Why Use Streams?
3. Types of Streams
4. Using Readable Streams
5. Using Writable Streams
6. Duplex and Transform Streams
7. What Is a Buffer in Node.js?
8. Working with Buffers
9. Streams vs Buffers
10. Practical Use Case Example
11. Best Practices
12. Conclusion

1. What Are Streams in Node.js?

A Stream is an abstract interface for working with streaming data in Node.js. It allows data to be processed piece by piece, rather than loading everything into memory at once.

Common use cases include:

• Reading/writing files
• Handling HTTP requests and responses
• Processing video/audio data
• Reading large CSVs or logs

2. Why Use Streams?

Streams are memory-efficient and non-blocking. For example, reading a 2 GB file with traditional file handling methods may crash the system due to memory overload. Streams handle such tasks efficiently by reading chunks of data progressively.

3. Types of Streams

Node.js provides four fundamental types of streams:

• Readable – Data can be read from them (fs.createReadStream)
• Writable – Data can be written to them (fs.createWriteStream)
• Duplex – Both readable and writable (net.Socket)
• Transform – Modify or transform the data while streaming (zlib.createGzip)

4. Using Readable Streams

To read from a file using streams:

const fs = require('fs');

const readable = fs.createReadStream('largefile.txt', { encoding: 'utf8' });

readable.on('data', (chunk) => {
  console.log('Received chunk:', chunk);
});

readable.on('end', () => {
  console.log('Finished reading file.');
});

5. Using Writable Streams

To write data to a file:

const fs = require('fs');

const writable = fs.createWriteStream('output.txt');

writable.write('Hello World\n');
writable.write('Streaming data to file.\n');
writable.end('Done writing.');

6. Duplex and Transform Streams

A Duplex Stream allows both read and write operations.

const { Duplex } = require('stream');

const duplex = new Duplex({
  read(size) {
    this.push('Data from duplex stream\n');
    this.push(null); // Ends stream
  },
  write(chunk, encoding, callback) {
    console.log('Writing:', chunk.toString());
    callback();
  }
});

duplex.on('data', (chunk) => {
  console.log('Read:', chunk.toString());
});

duplex.write('Hello Duplex!');

Transform Streams are a subtype of duplex that modifies data:

const { Transform } = require('stream');

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

process.stdin.pipe(upperCaseTransform).pipe(process.stdout);

7. What Is a Buffer in Node.js?

A Buffer is a temporary memory storage used to hold binary data. It’s particularly useful when dealing with streams, file systems, or binary protocols.

const buffer = Buffer.from('Hello Node.js');
console.log(buffer); // <Buffer 48 65 6c 6c 6f 20 4e 6f 64 65 2e 6a 73>
console.log(buffer.toString()); // Hello Node.js

8. Working with Buffers

Buffers are instances of the Buffer class:

// Allocating a buffer of size 10
const buf = Buffer.alloc(10);

// Writing to a buffer
buf.write('abc');
console.log(buf.toString()); // abc

You can manipulate buffers using methods like .slice(), .copy(), .length, etc.

9. Streams vs Buffers

10. Practical Use Case Example

Combining readable and writable streams to copy a file:

const fs = require('fs');

const reader = fs.createReadStream('input.txt');
const writer = fs.createWriteStream('output.txt');

reader.pipe(writer);

The pipe() method connects two streams, where readable data from the first is passed into the writable stream.

11. Best Practices

• Always handle error events on streams:

readable.on('error', (err) => console.error('Read error:', err));
writable.on('error', (err) => console.error('Write error:', err));

• Use pipe() for readable → writable connections to simplify code
• Use buffers for raw binary data like images or file manipulation
• Use stream backpressure techniques for high-performance applications

12. Conclusion

Streams and Buffers are at the core of many high-performance Node.js applications. Mastering these tools allows developers to efficiently manage memory and process large-scale data in chunks, instead of loading it all at once. Whether you’re building a media streaming service, data processor, or working with files and sockets, these tools will make your application robust and scalable.