Scraping a website involves implementing measures to avoid being detected as a bot. In this tutorial, we'll change the User Agent in Playwright in both Python and Node.js and cover best practices for doing so reliably.
What Is the Playwright User Agent?
The Playwright User Agent is a string that identifies you to the websites you visit when using the headless browser. It's typically included in the HTTP request header and shares information about your operating system and browser.
A typical User Agent from a real Chrome browser looks like this:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
From it, we can tell that, for example, the user is accessing the website using Chrome version 148 on a Windows 10.
However, the default Playwright UA is different and looks something like the one below, with the second part being the version:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/145.0.7632.6 Safari/537.36
The HeadlessChrome flag is an obvious bot signal that tells anti-bots that you're running an automated headless browser. Anti-bots use this and other techniques to detect and block web scrapers. Changing this default Playwright User Agent to that of a real browser is the minimum required step for scraping without being immediately flagged.
How to Set a Custom User Agent in Playwright
Let's learn how to set a custom User Agent (UA) in Playwright to scrape the web while flying under the radar!
Set Playwright User Agent in Python
First, install dependencies using pip and install the Chromium binary:
pip3 install playwright
playwright install chromium
The userAgent parameter goes on browser.new_context(). Every page opened in that context inherits the custom UA:
# pip3 install playwright
# playwright install chromium
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
# create a new browser context with a custom user agent
context = browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36"
)
page = context.new_page()
page.goto("https://httpbin.io/user-agent")
print(page.content())
browser.close()
And here's the output of the above code:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36
Change Playwright User Agent in Node.js
Install the dependencies:
npm install playwright
npx playwright install chromium
Then pass the same userAgent parameter on browser.newContext:
const { chromium } = require('playwright');
(async () => {
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext({
userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36',
});
const page = await context.newPage();
await page.goto('https://httpbin.io/user-agent');
const content = await page.innerText('body');
console.log(content);
await browser.close();
})();
We used Httpbin.io as a target URL to get our UA in the response. We've only used it as a test endpoint to displays the default User Agent in Playwright.
If you want to review your fundamentals, take a look at our Playwright tutorial.
sec-ch-ua Consistency
Setting a real User Agent isn't enough on its own. Anti-bot systems also check the sec-ch-ua Client Hints headers that your browser sends along with the User Agent string. This header also references the User Agent version, platform, and vendor, and these values must be consistent with those of the User Agent.
The Client Hints headers typically look like this:
"sec-ch-ua": '"Chromium";v="148", "Google Chrome";v="148", "Not/A)Brand";v="99"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
So, if your User Agent says Chrome 148 but the sec-ch-ua header is absent or references a different version or vendor, that mismatch is a detection signal.
When setting a custom UA, it's best to also set the corresponding Client Hints headers:
# pip3 install playwright
# playwright install chromium
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
# create a new browser context with custom user agent and Client Hints headers
context = browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
extra_http_headers={
"sec-ch-ua": '"Chromium";v="148", "Google Chrome";v="148", "Not/A)Brand";v="99"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": '"Windows"',
},
)
page = context.new_page()
page.goto("https://httpbin.io/headers")
print(page.inner_text("body"))
browser.close()
It's important to keep the browser version consistent across user_agent and sec-ch-ua. If you rotate UA strings, rotate the Client Hints headers in sync as well.
How to Rotate User Agents in Playwright
Using a single UA across many requests creates a pattern that anti-bot systems can flag over time. Rotating through a list of realistic UA strings makes your traffic look like it's coming from different users.
To rotate the User Agent, first create a list of UAs. We'll take some examples from our list of User Agents for web scraping. Here's how to add them:
# ...
# list of realistic user agents to rotate through
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
]
Here's how to rotate Playwright User Agents in Python:
# pip3 install playwright
# playwright install chromium
import random
from playwright.sync_api import sync_playwright
# list of user agents to rotate through
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
]
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
# create a new browser context with a random user agent
context = browser.new_context(user_agent=random.choice(USER_AGENTS))
page = context.new_page()
page.goto("https://httpbin.io/user-agent")
print(page.inner_text("body"))
browser.close()
random.choice() picks a random item from the list each time the script runs, so each execution uses a different UA.
And for Node.js:
// npm install playwright
// npx playwright install chromium
const { chromium } = require('playwright');
// list of user agents to rotate through
const userAgentStrings = [
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36',
];
(async () => {
const browser = await chromium.launch({ headless: true });
// create a new browser context with a random user agent
const context = await browser.newContext({
userAgent: userAgentStrings[Math.floor(Math.random() * userAgentStrings.length)],
});
const page = await context.newPage();
await page.goto('https://httpbin.io/user-agent');
console.log(await page.innerText('body'));
await browser.close();
})();
Math.random() generates a floating-point number between 0 and 1. Multiplying it by the list length and passing it to Math.floor() gives a random integer index, which selects a UA from the array.
However, When scraping multiple URLs in a single run, the single-context pattern above won't actually rotate because every page in the session uses the User Agent set at launch. To rotate per request, create a new context for each URL:
# pip3 install playwright
# playwright install chromium
import random
from playwright.sync_api import sync_playwright
# list of user agents to rotate through
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.0.0 Safari/537.36",
]
urls = [
"https://httpbin.io/user-agent",
"https://httpbin.io/user-agent",
]
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
# rotate user agents for each context
for url in urls:
# create a new browser context with a random user agent
context = browser.new_context(user_agent=random.choice(USER_AGENTS))
page = context.new_page()
page.goto(url)
print(page.inner_text("body"))
context.close()
browser.close()
Each iteration opens a fresh context with a newly selected UA, then closes it before the next request. This is the pattern to use when you need genuine per-request rotation during a continuous loop for large scale scraping.
It's important to keep your User Agent list up-to-date to avoid being flagged as a bot. However, this can be a tiring process and somewhat difficult to keep up with.
Best Practices for Setting User Agent in Playwright
- Keep UA strings current: Anti-bot systems maintain fingerprint databases and stale browser like these versions raise suspicion. Always use a recent browser version in your User Agent.
- Match OS and browser version: A Chrome 148 User Agent string with a Windows OS identifier paired with macOS-specific headers is a mismatch. Keep the OS consistent across the UA string and any associated headers.
- Keep
sec-ch-uain sync: If you rotate UA strings, rotate the Client Hints headers in the same operation. Mismatches between the UA and sec-ch-ua are a common detection trigger on systems like DataDome and Akamai. - User Agent rotation alone isn't enough: Modern anti-bot systems check IP reputation, TLS fingerprints, browser automation flags, and behavioral patterns alongside the User Agent. Rotating UAs addresses only one signal. Meaning it won't bypass Cloudflare or DataDome on its own.
Change User Agents in Playwright At Scale
Having a reliable User Agent rotation system requires constant attention. Beyond just creating a list, you need to regularly check browser versions, make sure they match the correct operating systems, correctly referenced in the Client Hints, and clean up outdated combinations.
More importantly, using different User Agents alone won't stop websites from detecting your Playwright automation. Modern websites check many other factors, such as mouse movements, IP reputation, connection details, and more, to spot bots.
A better solution is to use the ZenRows Universal Scraper API, which handles User Agent management automatically, supports JavaScript rendering, auto-bypasses any anti-bot measure at scale. With it's Adaptive Stealth Mode, ZenRows automatically picks the best anti-bot bypass strategy for any specific website, minimizing cost and eliminating maintenance overhead.
Let's see how ZenRows performs on a protected page, such as the Antibot Challenge page.
Start by signing up for a new account and go to the ZenRows Playground. Insert the target URL, enable Adaptive Stealth Mode.
Next, choose your preferred language from the language dropdown, then click the API connection mode. Next, copy the generated code and paste it into your script.
Here's a sample code generated for Python:
# pip3 install requests
import requests
url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"mode":"auto",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
When you run this code, you'll successfully access the page:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Using ZenRows, you've accessed a protected page without any complex Playwright setup.
Conclusion
User Agent management is a necessary first step, not a complete solution. Setting a realistic Chrome User Agent removes the most obvious automation signal, and rotating it across requests reduces the risk of pattern-based detection. But modern anti-bot systems check far more than the User Agent string. They analyze IP reputation, TLS fingerprints, browser automation flags, and behavioral patterns.
If you're hitting blocks despite correct User Agent configuration, the problem is almost certainly one of those other layers. The best way to avoid getting blocked and shield your scraper against all detection layers with zero infrastructure or technical overhead is to use a scraping solution like ZenRows. Try ZenRows for free now or speak with sales!