Downloading a file from a remote server using NodeJs without any npm package

---

Table of Contents

---

Problem

• There exists a file on a remote web server
• We have the url of the file
• How can we download the file from a NodeJS application?

Pre-requisite

• An intermediate level understanding of NodeJS and Javascript
• You have to be familiar with the idea of streams. click here to learn about NodeJS streams

💡 The code is a gradual build up to the final working code, you can keep track of the curent section by paying attention to the comments within the code

Setup Requirements

First, we need to specify the file URL we want to download, we would store that in a variable called remoteURL. We also need to specify the path where the downloaded file will be saved, we would call that outputFilePath

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

Preparing the output stream

NodeJS built-infs module provides a method createWriteStream which takes in a file path as parameter and returns a writable stream object for us.

We would pass our variable outputFilePath as the path parameter and store the returned writable stream object in a variable called outputStream

const fs  = require("fs")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

//prepare output stream
const outputStream= fs.createWriteStream(outputFilePath)

Preparing the incoming stream

To create a readable stream from the remote URL we would utilize the NodeJS built-in http module (use the https module if your url uses https protocol. the result is the same except you can be sure the transport was secure)

The http or https module provides a get method that takes in the URL and a callback function as parameters.

The get function will contact the remote server using the URL we provide. Once the response is ready, it will call the callback function we provide and pass it the response object that we can get our file from.

The get method returns a request object which will come in handy for handling errors.

💡 We could also use third party libraries like node-fetch or isomorphic-unfetch instead of the built-in http module. They will all return a response stream we can work with

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

// Preparing incoming stream
const request=http.get(remoteURL,(response)=>{
});

Download file

To download the file, all we have to do is call the pipe method on the response object which is a readable stream.

The pipe method takes in a writable stream as parameter.

We already have a writable stream which is our file outputStream above.

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

const request=http.get(remoteURL,(response)=>{
    //Download file
    response.pipe(outputStream)
});

Handle Download Complete

All streams in NodeJS are also event emitters so we can listen to certain events with the on method provided.

To know when our download is complete we would listen for the finish event on our outputStream.

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

const request=http.get(remoteURL,(response)=>{
    response.pipe(outputStream)
});

//Handle download complete
outputStream.on("finish",()=>{
    console.log("download complete!")
})

Error Handling

Handle Request Errors

Remember when I said the http.get method returns a request object that will come in handy for handling errors?

Yea, the request object is an event emitter. We can listen for the error event. The callback we provide is called when an error occurs.

The error event passes an error object to our callback

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

const request=http.get(remoteURL,(response)=>{
    response.pipe(outputStream)
});

outputStream.on("finish",()=>{
    console.log("download complete!")
})

//Handle request error
request.on("error",(error)=>{
   console.error("request error",error)
})

Handle response errors

The response object we got from our http.get method has a property statusCode that contains the http status code of the response from the server.

From basic understanding of http protocol, we should be expecting the remote server to send us a status code 200(ok) if all went well. Anything else we will consider an error in this case.

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

const request=http.get(remoteURL,(response)=>{
 //Handle response error
 if(response.statusCode!=200) {
  console.error("response error", response.statusCode)
  return
 }
 response.pipe(outputStream)
});

outputStream.on("finish",()=>{
   console.log("download complete!")
})

request.on("error",(error)=>{
  console.error("request error",error)
})

Handle Download Errors

To handle any error that occurs during the download, we will pay attention to our outputStream. Our outputStream as we already know is an event emitter, so we can listen for the error event.

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

const request=http.get(remoteURL,(response)=>{
  if(response.statusCode!=200) {
   console.error("response error", response.statusCode)
   return
  }
  response.pipe(outputStream)
});

outputStream.on("finish",()=>{
    console.log("download complete!")
})

request.on("error",(error)=>{
   console.error("request error",error)
})

//Handle Download error
outputStream.on("error",(error)=>{
   console.error("download error",error)
})

Delete Partial Download

When an error occurs during our download, we might have downloaded some parts of our file. This will lead to broken file in our directory.

Its good practice for us to delete this broken file. So we will use the unlink method from fs module and provide it with our outputFilePath. This should delete the file.

const fs  = require("fs")
const http = require("http")

const remoteURL = "http://your-remote-url.com"
const outputFilePath = "/your/file/path.ext"

const outputStream= fs.createWriteStream(outputFilePath)

const request=http.get(remoteURL,(response)=>{
  if(response.statusCode!=200) {
   console.error("response error", response.statusCode)
   return
  }
  response.pipe(outputStream)
});

outputStream.on("finish",()=>{
    console.log("download complete!")
})

request.on("error",(error)=>{
   console.error("request error",error)
})

outputStream.on("error",(error)=>{
   console.error("download error",error)
  //Delete partial Download
   fs.unlink(outputFilePath,()=>{ 
      console.error("partial file deleted")
   })
})

How to run the code

1. save the code in index.js file
2. Run node index.js command in the directory where the index.js file is located

Live Demo

You can run the live demo on replit

Conclusion

I hope this explanation was helpful. I certainly could have made the code shorter by utilizing third party libraries.

But I fear that would have taken away some fine details that those libraries will abstract.

So this explanation became a good lesson on streams, events, and good code practices. Who would have thought?

See you later. Bye 👋🏾