A major issue when scraping with a tool like Puppeteer is getting blocked by your target website. If you've encountered this, it's likely that your User Agent flagged you as a bot.
In this article, you'll learn how to change the Puppeteer User Agent to avoid detection and access the data you need.
What Is the Puppeteer User Agent?
To understand what the Puppeteer User Agent is, you must know that HTTP request headers are a set of data that provide additional information about the request. They include details that help the server understand and process the request appropriately.
One of the key elements of an HTTP request header is the User Agent (UA), which provides information that helps the server identify clients and tailor its response.
Below is what a sample of real Google Chrome UA looks like:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36
There, Mozilla/5.0
is a product token that tells the server that the client is Mozilla-compatible. (Windows NT 10.0; Win64; x64)
means that the client runs on a 64-bit version of Windows 10, and AppleWebKit/537.36 (KHTML, like Gecko)
is the rendering engine with version number 537.36 that powers the client. Lastly, Chrome/111.0.0.0
and Safari/537.36
indicate that a Chrome browser is used and is compatible with Safari.
Now, let's see what your default Puppeteer User Agent looks like:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/116.0.5845.96 Safari/537.36
You can tell it doesn't look like a real one, so your requests are easily identifiable as automated.
How Do I Set a Custom User Agent in Puppeteer?
Let's customize your User Agent to bypass CAPTCHAs in Puppeteer. You will begin by installing Puppeteer on your local machine, then set a custom UA using a dedicated method, and finally randomize the UA.
Step 1. Getting Started
Let's set up a new Puppeteer project.
Start by installing Puppeteer if it's not installed yet:
npm install puppeteer
Then, initialize a new project:
npm init esnext
Here's a minimal script that makes a request to HTTPBin to log the User Agent of the requesting client:
import puppeteer from 'puppeteer';
(async () => {
// Launch the browser and open a new blank page
const browser = await puppeteer.launch({
headless: 'new',
});
const page = await browser.newPage();
// Navigate the page to target website
await page.goto('https://httpbin.io/user-agent');
// Get the text content of the page's body
const content = await page.evaluate(() => document.body.textContent);
// Log the text content
console.log('Content: ', content);
// Close the browser
await browser.close();
})();
Run the script, and you'll get an output like this:
{
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
}
This is your default UA.
You can have a look at our guide on web scraping with Puppeteer guide to review the fundamentals if needed.Â
Now, let's set a custom User Agent in Puppeteer.
Step 2. Customize UA
To customize your default UA, use the page.setUserAgent()
method, which takes in the new User Agent that you want to set as its first argument. Here's how:
// ...
const page = await browser.newPage();
// Custom user agent
const customUA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36';
// Set custom user agent
await page.setUserAgent(customUA);
// Navigate the page to target website
await page.goto('https://httpbin.io/user-agent');
// ...
Note that page.setUserAgent()
must be called immediately before navigating to your target website.
Your full Puppeteer script should look like this now:
import puppeteer from 'puppeteer';
(async () => {
// Launch the browser and open a new blank page
const browser = await puppeteer.launch({
headless: 'new',
});
const page = await browser.newPage();
// Custom user agent
const customUA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36';
// Set custom user agent
await page.setUserAgent(customUA);
// Navigate the page to target website
await page.goto('https://httpbin.io/user-agent');
// Get the text content of the page's body
const content = await page.evaluate(() => document.body.textContent);
// Log the text content
console.log(content);
// Close the browser
await browser.close();
})();
Run it, and you will get this output:
{
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36"
}
Awesome! The UA in the output is the same as the one you set in the code.
However, using a single custom UA is not enough to avoid getting blocked because you can still be easily identified when using the same UA many times. Therefore, you need to randomize your UA per request. Let's see how in the next section.
Step 3. Use a Random User Agent in Puppeteer
Clients that use the same UA to make too many requests may be denied access. Therefore, it's advisable to use a random User Agent with Puppeteer to avoid getting blocked.
To randomly rotate your it, first collect a pool of UAs to pick from. You can grab a few from our User Agent list for web scraping.
Now, let's create a separate function just below the Puppeteer import that returns a different UA whenever it's called.
// ...
const generateRandomUA = () => {
// Array of random user agents
const userAgents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
];
// Get a random index based on the length of the user agents array
const randomUAIndex = Math.floor(Math.random() * userAgents.length);
// Return a random user agent using the index above
return userAgents[randomUAIndex];
}
// ...
Adding the function generateRandomUA()
to the main Puppeteer code, you'll have:
// ...
// Open a new blank page
const page = await browser.newPage();
// Custom user agent from generateRandomUA() function
const customUA = generateRandomUA();
// Set custom user agent
await page.setUserAgent(customUA);
// ...
This is your full code:
import puppeteer from 'puppeteer';
const generateRandomUA = () => {
// Array of random user agents
const userAgents = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15'
];
// Get a random index based on the length of the user agents array
const randomUAIndex = Math.floor(Math.random() * userAgents.length);
// Return a random user agent using the index above
return userAgents[randomUAIndex];
}
(async () => {
// Launch the browser
const browser = await puppeteer.launch({
headless: 'new',
});
// Open a new blank page
const page = await browser.newPage();
// Custom user agent from generateRandomUA() function
const customUA = generateRandomUA();
// Set custom user agent
await page.setUserAgent(customUA);
// Navigate the page to target website
await page.goto('https://httpbin.io/user-agent');
// Get the text content of the page's body
const content = await page.evaluate(() => document.body.textContent);
// Log the text content
console.log(`Request ${i + 1}: `, content);
// Close the page
await page.close();
// Close the browser
await browser.close();
})();
Run it twice, and you'll get a randomly selected Puppeteer User Agent logged each time. For example, we got this:
// Output 1
{
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Safari/605.1.15"
}
// Output 2
{
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
}
Great! Your User Agent in Puppeteer now changes per request so that you have more chances to do web scraping without getting blocked.
To be noted, when adding UAs, you must ensure that they're well constructed to prevent you from being flagged as a bot. Examples of incorrect ones include those that are outdated or have inconsistencies between the browser name and published versions.
Yet, maintaining a pool of valid UAs yourself is hard. Luckily, there's an easier solution we'll learn about in the next section.
Easy Solution to Change User Agent in Puppeteer at Scale
Creating a good User Agent rotation system isn't simple. You need to keep your list updated with new browser versions, make sure they match operating systems correctly and remove old combinations.
Besides, websites look at more than just User Agents to detect bots. They check your browsing patterns, network behavior, IP reputation, and many other technical details. Even with perfect User Agent rotation, your Puppeteer scripts might still get blocked.
Instead of managing all this yourself, use ZenRows' Universal Scraper API. It handles User Agents automatically, supports JavaScript rendering, auto-bypasses any CAPTCHA, and provides you with everything you need to avoid getting blocked.
Let's see how ZenRows performs against a protected page like the Antibot Challenge page.
Start by signing up for a new account to get to the Request Builder.

Insert the target URL, enable JS Rendering, and activate Premium Proxies.
Next, choose Node.js and then click on the API connection mode. After that, copy the generated code and paste it into your script.
const axios = require('axios');
const params = {
url: 'https://www.scrapingcourse.com/antibot-challenge',
apikey: '<YOUR_ZENROWS_API_KEY>',
js_render: 'true',
premium_proxy: 'true'
};
axios.get('https://api.zenrows.com/v1/', { params })
.then(({ data }) => console.log(data))
.catch(error => console.error(error));
The generated code uses the Axios library as the HTTP client. You can install this library using the following command:
npm install axios
When you run this code, you'll successfully access the page:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
Congratulations! 🎉 You've successfully accessed a protected page without any complex Puppeteer configuration.
Conclusion
In this guide, you've learned important things about User Agents in Puppeteer:
- How User Agents work in Puppeteer.
- Ways to set custom User Agents.
- How to rotate between different User Agents.
- Why User Agent management isn't enough by itself.
Keep in mind that Puppeteer is not the best solution for avoiding anti-bot systems. Use ZenRows to make sure you extract all the data you need without getting blocked. Try ZenRows for free!