Does your Playwright scraper keep hitting a roadblock? You're not alone. Playwright fingerprinting plays a significant role in how websites detect and block you.
In this article, you'll learn how to improve Playwright's fingerprints to boost its stealth against anti-bot measures. Let's get started!
What Is Playwright Fingerprinting?
Playwright fingerprinting describes how websites identify Playwright's automation. Although not a standard term, it's part of the browser fingerprinting techniques that distinguish automated browsers from real users.
Properties such as the IP address, browser configurations, plugins, graphic renderers, hardware concurrency, screen resolution, and others contribute to a browser's fingerprint. Even the slightest variation in these attributes can make a client distinguishable.
Why Playwright Fingerprinting Is a Threat to Web Scraping?
During browser fingerprinting, anti-bots often compare a client's fingerprints with those of known bots. Once it matches that of a known bot, the website can implement measures to block it from accessing its content.
Unfortunately, scraping with Playwright exposes bot-like fingerprints, such as the HeadlessChrome flag in headless mode, webdriver.navigator=true, bot-like ShiftShader WebGL renderer, and more. Although anti-bots like Cloudflare, PerimeterX, and DataDome may not directly fingerprint Playwright, their bot detection measures are sensitive to these signals.
One way to avoid detection in Playwright is to manually tweak these inherent bot-like properties to resemble a real browser's fingerprints. However, this manual approach is often technical and time-consuming. A better solution is to spoof these fingerprints automatically with third-party helpers and plugins like playwright-with-fingerprints.
The next sections cover how this plugin works and how to apply it to reduce the chances of being detected during scraping.
What Is playwright-with-fingerprints and How Does it Work?
playwright-with-fingerprints is a plugin designed to increase Playwright's stealth by spoofing actual browser fingerprints with minimal effort. Under the hood, it uses the FingerprintSwitcher, which lets you replace suspicious browser properties.
The plugin mimics a real browser's runtime, even in headless mode. It achieves this by patching automation properties, such as changing the value of webdriver.navigator to true, switching the WebGL property, and replacing the HeadlessChrome flag with Chrome.
Let's apply this fingerprinting plugin in Playwright!
Bypass Browser Fingerprinting With playwright-with-fingerprints During Scraping
In this section, you'll use the playwright-with-fingerprints plugin to modify Playwright's properties. You'll then test its effect on SannySoft, a browser fingerprinting testing platform.
First, install the plugin using npm. Note that the tool works alongside the standard Playwright library. So you need to install them together:
npm install playwright playwright-with-fingerprints
The playwright-with-fingerprints plugin only works on Windows operating systems. Installing and running it on unsupported platforms might result in errors.
Before applying the plugin, run the following script to see how Playwright's default fingerprints appear. Launch a Playwright browser instance and open the test website. Then, take a screenshot of the test result:
// npm install playwright
// npx playwright install
const { chromium } = require('playwright');
(async () => {
// launch a Chromium browser instance
const browser = await chromium.launch();
// open a new page
const page = await browser.newPage();
await page.goto('https://bot.sannysoft.com/');
// take a screenshot
await page.screenshot({ path: 'screenshot.png' });
// close the browser
await browser.close();
})();
The default Playwright library fails major fingerprinting tests, as shown:
Let's improve that result with the playwright-with-fingerprints plugin.
Step 1: Configure the Fingerprint Service
Starting the fingerprint service requires replacing the base Playwright with the playwright-with-fingerprints plugin. The tool inherits Playwright's API under the hood, giving you an optimized headless browser.
First, import the playwright-with-fingerprints library and instantiate its service using an empty key. The empty string in setServiceKey means you're using the free service of the FingerprintSwitcher, which is enough for basic spoofing.
/ npm install playwright playwright-with-fingerprints
// npx playwright install
const { plugin } = require('playwright-with-fingerprints');
// instantiate a plugin service
plugin.setServiceKey('');
You've set the fundamentals. It's time to apply specific fingerprints.
Step 2: Fetch and Apply Fingerprints
The next step is to fetch a browser fingerprint from the FingerprintSwitcher database.
Set the plugin to use the Chrome browser on a Windows operating system. Then, use it to launch a Chrome instance:
// ...
(async () => {
// fetch a browser fingerprint from the server
const fingerprint = await plugin.fetch({
tags: ['Microsoft Windows', 'Chrome'],
});
// add the fingerprints
plugin.useFingerprint(fingerprint);
// launch a Chromium browser instance with fingerprints
const browser = await plugin.launch();
})();
The above modification configures your Playwright scraper to use an actual Chrome fingerprint. In the next section, you'll test the setup by requesting the test site.
Spoofing of platforms other than Windows OS is only available in the paid version of the tool.
Step 3: Test Fingerprint Behavior
Let's now test if the new fingerprint makes a difference. Open the SannySoft website and screenshot the result. To achieve this, add the following to the previous code:
// ...
(async () => {
// ...
// open a new page
const page = await browser.newPage();
await page.goto('https://bot.sannysoft.com/');
// take a screenshot
await page.screenshot({ path: 'screenshot.png' });
// close the browser
await browser.close();
})();
Combine the snippets, and the final code looks like this:
// npm install playwright playwright-with-fingerprints
// npx playwright install
const { plugin } = require('playwright-with-fingerprints');
// instantiate a plugin service
plugin.setServiceKey('');
(async () => {
// fetch a browser fingerprint from the server
const fingerprint = await plugin.fetch({
tags: ['Microsoft Windows', 'Chrome'],
});
// add the fingerprints
plugin.useFingerprint(fingerprint);
// launch a Chromium browser instance with fingerprints
const browser = await plugin.launch();
// open a new page
const page = await browser.newPage();
await page.goto('https://bot.sannysoft.com/');
// take a screenshot
await page.screenshot({ path: 'screenshot.png' });
// close the browser
await browser.close();
})();
The code returns the following result, showing that Playwright passes all the fingerprinting tests after using the plugin:
That's it! You've modified your scraper with an actual browser fingerprint and are now better equipped to bypass Playwright fingerprinting.
Yet, patching bot-like properties with the playwright-with-fingerprints plugin has some limitations. We'll discuss it next.
Limitations of the playwright-with-fingerprints Plugin
While the playwright-with-fingerprints plugin spoofs actual browser fingerprints, it doesn't cover all detection methods.
In addition to browser fingerprinting, websites use several other anti-scraping techniques to detect bots. These include behavioral analysis, rate limiting, request header monitoring, and JavaScript challenges, among others. You can't tell what you're up against at any point.
It might also introduce suspicious fingerprints. For example, the User Agent header in the previous patch is outdated and might get flagged. All these limitations make the plugin ineffective against advanced anti-bot measures.
For example, the plugin fails against a well-protected website like this Antibot Challenge page. Try it out by replacing the previous test URL with this new one, as shown in the code below:
// npm install playwright playwright-with-fingerprints
// npx playwright install
const { plugin } = require('playwright-with-fingerprints');
// instantiate a plugin service
plugin.setServiceKey('');
(async () => {
// fetch a browser fingerprint from the server
const fingerprint = await plugin.fetch({
tags: ['Microsoft Windows', 'Chrome'],
});
// add the fingerprints
plugin.useFingerprint(fingerprint);
// launch a Chromium browser instance with fingerprints
const browser = await plugin.launch();
// open a new page
const page = await browser.newPage();
await page.goto('https://www.scrapingcourse.com/antibot-challenge');
// take a screenshot
await page.screenshot({ path: 'screenshot.png' });
// close the browser
await browser.close();
})();
Playwright got blocked despite patching it with the playwright-with-fingerprints plugin:
The above result proves that spoofing real browser fingerprints isn't enough against advanced anti-bot measures.
The good news is that there is a solution to these limitations. The next section will explain it.
The Best Scraping Techniques to Avoid Detection
The plugin already enhances Playwright with some decent fingerprints, but as you've seen, those are insufficient. Another decent improvement is to use a proxy with Playwright. However, even that isn't enough against complex anti-bot measures.
A web scraping API, such as ZenRows' Universal Scraper API, is the best way to bypass detection reliably. ZenRows is an all-in-one scraping solution that simplifies your workflow and allows you to access any website without being blocked.
A single API call to ZenRows fortifies your scraper with advanced fingerprints, optimized request headers, premium rotating proxies, JavaScript rendering support, cutting-edge anti-bot auto-bypass technology, and everything else you need for reliable web scraping. It also has headless browser features for executing human interactions, making it an excellent alternative to a more resource-intensive headless browser like Playwright.
Let's see how ZenRow works by scraping the Anti-bot Challenge page that previously blocked your plugin-fortified Playwright scraper.
Sign up to open the ZenRows Request Builder. Paste the target URL in the link box and activate Premium Proxies and JS Rendering.
Then, choose Node.js as your programming language and select the API connection mode. Copy and paste the generated code into your crawler file:
The generated code looks like this:
// npm install axios
const axios = require('axios');
const url = 'https://www.scrapingcourse.com/antibot-challenge';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
url: 'https://api.zenrows.com/v1/',
method: 'GET',
params: {
url: url,
apikey: apikey,
js_render: 'true',
premium_proxy: 'true',
},
})
.then((response) => console.log(response.data))
.catch((error) => console.log(error));
The code accesses the protected website and outputs its full-page HTML, proving ZenRows works:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! 🎉 You just used ZenRows to bypass an anti-bot measure.
Conclusion
In this tutorial, you've learned how Playwright fingerprinting contributes to anti-bot detection and how to bypass it with the playwright-with-fingerprints plugin.
Despite being a powerful tool, the plugin isn't enough against advanced anti-bot systems. We recommend entirely fortifying your scraper with ZenRows. It provides all the toolkits required to scrape any website at scale without getting blocked.