Handle the User Agent in NodeJS

When scraping with NodeJS, a default or bad User Agent is one of the most frequent reasons why you get blocked. So here, you'll learn how to set a custom one and randomize it using Axios in NodeJS to ensure the success of your project.

What Is the User-Agent in NodeJS?

The User-Agent header is a string sent with an HTTP request, identifying the client making the request. In NodeJS, you can use a library such as Axios or Selenium to make HTTP requests.

Install Axios with the following command:

npm install axios

To see your current UA, send a request to https://httpbin.org/headers and log the User-Agent header using Axios. You can copy-paste the below sample scraper and save it in a file named index.js, for example.

const axios = require('axios');
 
axios.get('https://httpbin.org/headers')
  .then(response => {
    const headers = response.data.headers;
    console.log(headers['User-Agent']);
  })
  .catch(error => {
    console.log(error);
  });

To run it, use the node command followed by your file name.

As you see, the default NodeJS User Agent using Axios is axios/1.4.0. (it can be slightly different for you depending on the library version you use). This default UA string is easily identifiable by website security measures and will likely get you blocked.

Let's learn how to change it!

How to Use a Custom User-Agent in NodeJS

To set a custom User Agent in NodeJS, follow this process:

The first step is to get a real sample, and you can find some reliable ones in our list of UAs for web scraping. Grab a User Agent and put it on an object:

const headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
};

You'll use this object as the second parameter for the Axios GET call:

axios.get('https://httpbin.org/headers', { headers })

So this is the complete code that sends a request to https://httpbin.com/headers with a custom User-Agent header and logs the User-Agent header received in the response:

const axios = require('axios');
 
const headers = {
  'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
};
 
axios.get('https://httpbin.org/headers', { headers })
  .then(response => {
    const headers = response.data.headers;
    console.log(headers['User-Agent']);
  })
  .catch(error => {
    console.log(error);
  });

After running the code, you should see an output similar to the one below:

Cool! You've just set a NodeJS fake User Agent.

However, more than a single User-Agent header is necessary to avoid getting blocked because too many consecutive requests with the same one are a hint to detect you as a bot. That's why you should rotate UA headers to use a different one for each request. 

Here's an example that sends a request to https://httpbin.org/headers with a randomly selected User Agent header. This code defines an array of User-Agents, chooses one randomly, and then uses a self-invoking async function to request the target website with the selected string.

const axios = require('axios');
const userAgents = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
];
 
(async function fetchWithRandomUserAgent() {
  const randomIndex = Math.floor(Math.random() * userAgents.length);
  const randomUserAgent = userAgents[randomIndex];
 
  const options = {
    headers: {
      'User-Agent': randomUserAgent
    }
  };
 
  try {
    const response = await axios.get('https://httpbin.org/headers', options);
    console.log(response.data);
  } catch (error) {
    console.error(error);
  }
})();

The script also logs the random User Agent from the response data to the console, so it worked!

How to Change the User-Agent in NodeJS at Scale

Setting up a reliable User Agent rotation system is more complex than managing a simple array. You need to regularly update browser versions, make sure they match with operating systems correctly and remove outdated combinations.

Plus, websites check more than just User Agents to detect bots. They analyze your request patterns, header consistency, browser behavior, and more. Even with perfect User Agent rotation, your requests might still get blocked.

A better solution is to use the ZenRows' Universal Scraper API. It automatically manages User Agents, handles all browser details, does JavaScript rendering, auto-bypasses any CAPTCHA, and provides you with everything you need to avoid getting blocked.

Let's test ZenRows with a website that usually blocks Axios requests like the Antibot Challenge page.

Start by signing up for a new account, to get to the Request Builder.

const axios = require('axios'); 

const params = { 
url: 'https://www.scrapingcourse.com/antibot-challenge', 
apikey: '<YOUR_ZENROWS_API_KEY>', 
js_render: 'true', 
premium_proxy: 'true' 
}; 

axios.get('https://api.zenrows.com/v1/', { params }) 
.then(({ data }) => console.log(data)) 
.catch(error => console.error(error));

The generated code uses the Axios library as the HTTP client. You can install this library using the following command:

npm install axios

When you run this code, you'll successfully access the page:

<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉 You’ve successfully bypassed the anti-bot challenge page using ZenRows. This works for any website.

Conclusion

This guide has shown you important things about User Agents in Node.js:

• What Is the User-Agent in NodeJS.
• How to use a custom User-Agent in NodeJS.
• How to change the User-Agent in NodeJS at scale.
• Why User Agent management alone isn't enough.

Remember, many websites use different anti-bot mechanisms to prevent web scraping. Integrate ZenRows to make sure you extract all the data you need without getting blocked. Try ZenRows for free!