Web Crawling Webinar for Tech Teams
Web Crawling Webinar for Tech Teams

How to Set Puppeteer User Agent

Idowu Omisola
Idowu Omisola
Updated: January 28, 2025 · 5 min read

A major issue when scraping with a tool like Puppeteer is getting blocked by your target website. If you've encountered this, it's likely that your User Agent flagged you as a bot.

In this article, you'll learn how to change the Puppeteer User Agent to avoid detection and access the data you need.

What Is the Puppeteer User Agent?

To understand what the Puppeteer User Agent is, you must know that HTTP request headers are a set of data that provide additional information about the request. They include details that help the server understand and process the request appropriately.

One of the key elements of an HTTP request header is the User Agent (UA), which provides information that helps the server identify clients and tailor its response.

Below is what a sample of real Google Chrome UA looks like:

Example
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36

There, Mozilla/5.0 is a product token that tells the server that the client is Mozilla-compatible. (Windows NT 10.0; Win64; x64) means that the client runs on a 64-bit version of Windows 10, and AppleWebKit/537.36 (KHTML, like Gecko) is the rendering engine with version number 537.36 that powers the client. Lastly, Chrome/111.0.0.0 and Safari/537.36 indicate that a Chrome browser is used and is compatible with Safari.

Now, let's see what your default Puppeteer User Agent looks like:

Example
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/116.0.5845.96 Safari/537.36

You can tell it doesn't look like a real one, so your requests are easily identifiable as automated.

Frustrated that your web scrapers are blocked once and again?
ZenRows API handles rotating proxies and headless browsers for you.
Try for FREE

How Do I Set a Custom User Agent in Puppeteer?

Let's customize your User Agent to bypass CAPTCHAs in Puppeteer. You will begin by installing Puppeteer on your local machine, then set a custom UA using a dedicated method, and finally randomize the UA.

Step 1. Getting Started

Let's set up a new Puppeteer project.

Start by installing Puppeteer if it's not installed yet:

Terminal
npm install puppeteer

Then, initialize a new project:

Terminal
npm init esnext

Here's a minimal script that makes a request to HTTPBin to log the User Agent of the requesting client:

scraper.js
import puppeteer from 'puppeteer';
 
(async () => {
  // Launch the browser and open a new blank page
  const browser = await puppeteer.launch({
    headless: 'new',
  });
  const page = await browser.newPage();
 
  // Navigate the page to target website
  await page.goto('https://httpbin.io/user-agent');
 
  // Get the text content of the page's body
  const content = await page.evaluate(() => document.body.textContent);
 
  // Log the text content
  console.log('Content: ', content);
 
  // Close the browser
  await browser.close();
})();

Run the script, and you'll get an output like this:

Output
{
  "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
}

This is your default UA.

Now, let's set a custom User Agent in Puppeteer.

Step 2. Customize UA

To customize your default UA, use the page.setUserAgent() method, which takes in the new User Agent that you want to set as its first argument. Here's how:

scraper.js
// ...
  const page = await browser.newPage();
 
  // Custom user agent
  const customUA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36';
 
  // Set custom user agent
  await page.setUserAgent(customUA);
 
  // Navigate the page to target website
  await page.goto('https://httpbin.io/user-agent');
 
// ...

Your full Puppeteer script should look like this now:

scraper.js
import puppeteer from 'puppeteer';
 
(async () => {
  // Launch the browser and open a new blank page
  const browser = await puppeteer.launch({
    headless: 'new',
  });
  const page = await browser.newPage();
 
  // Custom user agent
  const customUA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36';
 
  // Set custom user agent
  await page.setUserAgent(customUA);
 
  // Navigate the page to target website
  await page.goto('https://httpbin.io/user-agent');
 
  // Get the text content of the page's body
  const content = await page.evaluate(() => document.body.textContent);
 
  // Log the text content
  console.log(content);
 
  // Close the browser
  await browser.close();
})();

Run it, and you will get this output:

Output
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
}

Awesome! The UA in the output is the same as the one you set in the code.

However, using a single custom UA is not enough to avoid getting blocked because you can still be easily identified when using the same UA many times. Therefore, you need to randomize your UA per request. Let's see how in the next section.

Step 3. Use a Random User Agent in Puppeteer

Clients that use the same UA to make too many requests may be denied access. Therefore, it's advisable to use a random User Agent with Puppeteer to avoid getting blocked.

To randomly rotate your it, first collect a pool of UAs to pick from. You can grab a few from our User Agent list for web scraping.

Now, let's create a separate function just below the Puppeteer import that returns a different UA whenever it's called.

scraper.js
// ...
 
const generateRandomUA = () => {
  // Array of random user agents
  const userAgents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
  ];
  // Get a random index based on the length of the user agents array 
  const randomUAIndex = Math.floor(Math.random() * userAgents.length);
  // Return a random user agent using the index above
  return userAgents[randomUAIndex];
}
 
// ...

Adding the function generateRandomUA() to the main Puppeteer code, you'll have:

scraper.js
// ...
  // Open a new blank page
  const page = await browser.newPage();
 
  // Custom user agent from generateRandomUA() function
  const customUA = generateRandomUA();
 
  // Set custom user agent
  await page.setUserAgent(customUA);
// ...

This is your full code:

scraper.js
import puppeteer from 'puppeteer';
 
const generateRandomUA = () => {
  // Array of random user agents
  const userAgents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
  ];
  // Get a random index based on the length of the user agents array 
  const randomUAIndex = Math.floor(Math.random() * userAgents.length);
  // Return a random user agent using the index above
  return userAgents[randomUAIndex];
}
 
(async () => {
  // Launch the browser
  const browser = await puppeteer.launch({
    headless: 'new',
  });
 
  // Open a new blank page
  const page = await browser.newPage();
 
  // Custom user agent from generateRandomUA() function
  const customUA = generateRandomUA();
 
  // Set custom user agent
  await page.setUserAgent(customUA);
 
  // Navigate the page to target website
  await page.goto('https://httpbin.io/user-agent');
 
  // Get the text content of the page's body
  const content = await page.evaluate(() => document.body.textContent);
 
  // Log the text content
  console.log(`Request ${i + 1}: `, content);
 
  // Close the page
  await page.close();
 
  // Close the browser
  await browser.close();
})();

Run it twice, and you'll get a randomly selected Puppeteer User Agent logged each time. For example, we got this:

Output
// Output 1
{
  "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15"
}
 
// Output 2
{
  "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
}

Great! Your User Agent in Puppeteer now changes per request so that you have more chances to do web scraping without getting blocked.

To be noted, when adding UAs, you must ensure that they're well constructed to prevent you from being flagged as a bot. Examples of incorrect ones include those that are outdated or have inconsistencies between the browser name and published versions.

Yet, maintaining a pool of valid UAs yourself is hard. Luckily, there's an easier solution we'll learn about in the next section.

Easy Solution to Change User Agent in Puppeteer at Scale

Creating a good User Agent rotation system isn't simple. You need to keep your list updated with new browser versions, make sure they match operating systems correctly and remove old combinations.

Besides, websites look at more than just User Agents to detect bots. They check your browsing patterns, network behavior, IP reputation, and many other technical details. Even with perfect User Agent rotation, your Puppeteer scripts might still get blocked.

Instead of managing all this yourself, use ZenRows' Universal Scraper API. It handles User Agents automatically, supports JavaScript rendering, auto-bypasses any CAPTCHA, and provides you with everything you need to avoid getting blocked.

Let's see how ZenRows performs against a protected page like the Antibot Challenge page.

Start by signing up for a new account to get to the Request Builder.

building a scraper with zenrows
Click to open the image in full screen

Insert the target URL, enable JS Rendering, and activate Premium Proxies.

Next, choose Node.js and then click on the API connection mode. After that, copy the generated code and paste it into your script.

scraper.js
const axios = require('axios'); 

const params = { 
url: 'https://www.scrapingcourse.com/antibot-challenge', 
apikey: '<YOUR_ZENROWS_API_KEY>', 
js_render: 'true', 
premium_proxy: 'true' 
}; 

axios.get('https://api.zenrows.com/v1/', { params }) 
.then(({ data }) => console.log(data)) 
.catch(error => console.error(error));

The generated code uses the Axios library as the HTTP client. You can install this library using the following command:

Terminal
npm install axios

When you run this code, you'll successfully access the page:

Output
<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>

Congratulations! 🎉 You've successfully accessed a protected page without any complex Puppeteer configuration.

Conclusion

In this guide, you've learned important things about User Agents in Puppeteer:

  • How User Agents work in Puppeteer.
  • Ways to set custom User Agents.
  • How to rotate between different User Agents.
  • Why User Agent management isn't enough by itself.

Keep in mind that Puppeteer is not the best solution for avoiding anti-bot systems. Use ZenRows to make sure you extract all the data you need without getting blocked. Try ZenRows for free!

Ready to get started?

Up to 1,000 URLs for free are waiting for you