Streams in Node.js provide a powerful mechanism for handling data flow. They allow you to process data in small, manageable chunks, making them ideal for scenarios involving large files or network communication. In this article, we'll explore the concept of streams and demonstrate how to use them effectively in your Node.js applications.

In Node.js, there are four types of streams:

1. Readable Streams: Used for reading data from a source.

2. Writable Streams: Used for writing data to a destination.

3. Duplex Streams: Streams that are both readable and writable.

4. Transform Streams: Duplex streams that modify or transform the data as it passes through.

Using Streams to Read and Write Data

"Streams process data incrementally, which enables efficient handling of large files or network communication."

Let's explore how to use streams for reading and writing data in Node.js. Assume we have a file called sample.txt that we want to send to a server. Instead of reading the entire file into memory and sending it at once, we can use streams to process it in smaller chunks. But for now lets see an example of reading the file normally.

const express = require('express');
const fs = require('fs');
const statusMonitor = require('express-status-monitor');

const app = express();
const PORT = 8000;

// Enable Express Status Monitor middleware
app.use(statusMonitor());

app.get('/', (req, res) => {
  fs.readFile('./sample.txt', (err, data) => {
    if (err) {
      // Handle error if file reading fails
      console.error(err);
      res.status(500).send('Error reading file');
      return;
    }

    // Send the file data as the response
    res.end(data);
  });
});

app.listen(PORT, () => {
  console.log(`Server started at http://localhost:${PORT}`);
});

This code sets up an Express server, reads the contents of the "sample.txt" file, and sends the file data as the response. It also incorporates the Express Status Monitor middleware for monitoring the application's status and performance. If the "sample.txt" file is large, it may lead to increased memory usage and potential performance impacts due to loading the entire file into memory => SERVER CRASH!!

So here the memory usage is all time high which is not recommended.

"By processing data in smaller chunks, streams optimize memory usage and enable efficient handling of large files or network communication." Lets see how.

Streaming Data to the Browser with Express

"Streams can be leveraged to efficiently send data to the browser using Express.js and the res.write() method."

Now, let's see how we can use streams to efficiently send data to the browser using Express.js and the res.write() method. Assume we have an Express route that handles a request to view the file data.

const express = require('express');
const fs = require('fs');
const statusMonitor = require('express-status-monitor');

const app = express();

// Enable Express Status Monitor middleware
app.use(statusMonitor());

app.get('/', (req, res) => {
  const sourceFilePath = './sample.txt';
  const readStream = fs.createReadStream(sourceFilePath, 'utf8');

  // Stream data chunks to the response
  readStream.on('data', (chunk) => {
    res.write(chunk);
  });

  // Signal the end of the response
  readStream.on('end', () => {
    res.end();
  });
});

app.listen(8000, () => {
  console.log('Server started on port 8000');
});

Now you can see that memory usuage is only 55.5mb

In the above code, we create an Express route for the '/' endpoint. Within the route handler, we create a readable stream from the sample.txt file and listen for the 'data' event. As each chunk becomes available, we use res.write() to stream the data directly to the response. Finally, we call res.end() to indicate the end of the response.

Checking Response Header

The Transfer-Encoding: chunked response header indicates that the data is being transferred in chunks, enabling a continuous and efficient flow. This approach keeps the connection alive and allows for dynamic transmission without a predefined size, providing flexibility and responsiveness during the transfer process.

Example: Creating a zip file using zlib

zlib is a built-in Node.js module that provides compression and decompression functionality. It implements the Zlib compression algorithm, which is commonly used for file compression and data streaming.

const express = require('express');
const fs = require('fs');
const zlib = require('zlib');
const status = require('express-status-monitor');

const app = express();
const PORT = 8000;

app.use(status());

app.get('/', (req, res) => {
  // Stream Read (Sample.txt) --> Zipper --> fs Write Stream

  // Create a readable stream by reading the sample.txt file
  const readStream = fs.createReadStream('./sample.txt');

  // Pipe the data from the readable stream to the gzip compressor
  const gzip = zlib.createGzip();
  const compressedStream = readStream.pipe(gzip);

  // Pipe the compressed data to the writable stream to create the zip file
  const writeStream = fs.createWriteStream('./sample.zip');
  const writePipe = compressedStream.pipe(writeStream);

  // Handle events for completion and error
  writePipe.on('finish', () => {
    res.send('Zip file created successfully.');
  });

  writePipe.on('error', (err) => {
    console.error('Error creating the zip file:', err);
    res.status(500).send('Error creating the zip file.');
  });
});

app.listen(PORT, () => {
  console.log(`Server started on port ${PORT}`);
});

Monitoring Stream Usage with Express Status Monitor

"Express Status Monitor provides a visual representation of metrics, allowing you to monitor stream usage and application performance."

To monitor the stream usage and gain insights into the performance of your application, you can use Express Status Monitor. It provides a visual representation of various metrics, including memory usage, response time, and request rate. Let's integrate Express Status Monitor into our application.

const express = require('express');
const statusMonitor = require('express-status-monitor');
// Other required dependencies and configurations

const app = express();

app.use(statusMonitor());

// Your routes and middleware

app.listen(8000, () => {
  console.log('Server started on port 8000');
});

By adding the app.use(statusMonitor()) middleware, Express Status Monitor will be enabled for your application. You can then access the monitoring dashboard by visiting localhost:8000/status in your web browser.

Conclusion

Streams are a powerful feature in Node.js that allow you to efficiently process data in a chunk-by-chunk manner. They are particularly useful for scenarios involving large files or network communication. By using readable and writable streams, you can optimize memory usage and improve the performance of your applications.

In this article, we covered the basics of streams, demonstrated how to use them for reading and writing data, and showcased an example of streaming data to the browser using Express. Integrating streams into your applications can lead to more efficient and scalable data processing.

Thanks a lot for reading the article.
Hope you found it helpful.

Linkedin: https://www.linkedin.com/in/tautikk/
Email: tautikagrahari@gmail.com
Twitter: https://twitter.com/TautikA