Is your web scraper getting blocked even when using a headless browser? In this tutorial, you'll learn how to mask Playwright better to bypass Cloudflare.
- Approach 1: Use a stealth plugin.
- Approach 2: Use Proxies.
- Approach 3: Solve CAPTCHAs.
- Approach 4: Bypass Advanced Cloudflare Protections with ZenRows.
Let's begin!
What Is Cloudflare?
Cloudflare is a security and performance optimization company that offers bot management services. It blocks automated traffic, including web scraping operations. Cloudflare poses a significant challenge to web scraping due to its frequent firewall updates and the use of specialized honeypots, such as AI Labyrinth, to redirect bot traffic to the wrong destination.
During web scraping, you're more likely to encounter Cloudflare than most other web application firewalls (WAFs). In fact, W3Techs' 2025 market share report reveals that 80.8% of all websites that rely on a reverse proxy use Cloudflare.
Headless browsers, such as Playwright and Selenium, can't bypass Cloudflare security independently without some manual tweaks. So, if you're looking for a way to bypass Cloudflare with Playwright, you'll need additional strategies beyond the default setup.
How Cloudflare Works
Cloudflare employs various methods to distinguish between human users and bots. These include:
- Behavioral Analysis: Cloudflare tracks how users interact with the website, including their mouse movement patterns, cursor and typing speed, clicks, and page load times. Irregular behavior, such as repeated mouse movements, increased typing speed, and more, raises a red flag.
- Browser Fingerprinting: Cloudflare inspects the browser for specific attributes, such as the
window.navigatorproperty, User Agent, and runtime behavior. Although Playwright exhibits browser properties, it presents subtle irregularities in specific fields, such as the presence of the WebDriver in the navigator field or the indication of a headless browser runtime. These bot-like signals result in detection. - TLS Fingerprinting: Cloudflare also assesses a client's unique way of establishing secure connections during a TLS handshake. The TLS fingerprint reveals the internal request layer of each client, including the order of the TLS version, cipher suite, and extensions. Cloudflare leverages that information to differentiate between human and automated requests. For instance, an outdated TLS version can be flagged as a bot-like behavior.
- IP Reputation Analysis: Cloudflare also checks the IP of each request against a database of known IP addresses. If the scanned IP address matches that of a known bot flagged in previous scraping activities, Cloudflare blocks it. Another factor that lowers an IP's reputation is sending a large number of requests at once. Since bots often make multiple requests within a short time, Cloudflare easily detects and blocks their IP addresses.
- CAPTCHA Tests: When accessing a protected website, Cloudflare may decide to test if your request originates from a human or a robot using a challenge like Turnstile CAPTCHA. This usually appears on an interstitial page or while submitting a form. If the user passes, the security system grants them access to the target page. Otherwise, it blocks them. An automated browser like Playwright often fails this test since it can't solve or bypass the CAPTCHA by default.
Take a look at our Cloudflare bypass guide to learn more about these defensive techniques.
Why Base Playwright Is NOT Enough to Bypass Cloudflare
The base Playwright is insufficient against the previously mentioned Cloudflare detection methods because it lacks the internal evasion mechanisms to do so. This makes it unsuitable for scraping Cloudflare-protected websites.
Although Playwright and other browser automation libraries can simulate human-like browsing behavior and solve some initial JavaScript challenges, they fail against more advanced detection techniques. In that case, you may need to explore additional bypass techniques, such as using proxies, applying custom user agents, tweaking fingerprinting parameters, and more.
As proof, let's attempt to use Playwright to bypass the Cloudflare Challenge page protection and observe how it fails to evade Cloudflare. Try it out with the script below that screenshots the target page:
// npm install playwright
// npx playwright install
const { chromium } = require("playwright");
(async () => {
// launch the browser
const browser = await chromium.launch();
const page = await browser.newPage();
// open the target page and wait until the network is idle
await page.goto("https://www.scrapingcourse.com/cloudflare-challenge", {
waitUntil: "networkidle",
});
// screenshot the target page
await page.screenshot({ path: "screenshot.png", fullPage: true });
await browser.close();
})();
Here's the result:
Sadly, the base version of Playwright is recognized as a bot and consequently denied access to the website.
We'll explore four techniques that will help you win over Cloudflare in the next sections.
Approach 1: Use a Stealth Plugin
A plugin like Playwright-Extra with Puppeteer Stealth patches the bot-like loopholes in the standard Playwright library, reducing the likelihood of anti-bot detection. Specifically, Playwright Stealth modifies specific browser fingerprints, such as the following:
- Hiding Playwright's
navigator.webdriverattribute. - Changing the
HeadlessChromeflag in the User Agent to Chrome in headless mode. - Spoofing a real browser runtime to avoid detection in headless mode.
- Mimicking a real browser's plugins and extensions.
Let's see how to use the Playwright Stealth plugin with Puppeteer Extra. Start by installing the required libraries:
npm install playwright-extra puppeteer-extra-plugin-stealth
Now, import the libraries and add the Stealth plugin to your Playwright setup as shown:
// npm install playwright-extra playwright-extra-plugin-stealth playwright
const { chromium } = require("playwright-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth")();
// add the Stealth plugin to Chromium
chromium.use(StealthPlugin);
(async () => {
// set up the browser
const browser = await chromium.launch();
const page = await browser.newPage();
// open the target page
await page.goto("https://www.scrapingcourse.com/cloudflare-challenge");
const content = await page.content()
console.log(content)
await browser.close();
})();
However, while the Playwright-Extra plugin increases stealth, it still doesn't guarantee that you won't get blocked by Cloudflare's advanced detection mechanisms.
But no worries. Let's explore other methods that can help you bypass Cloudflare in Playwright.
Approach 2: Use Proxies
When scraping a website, it's easy to get banned if you make too many requests within a short period or use a previously blocked IP address. You can avoid that by rotating proxies to appear as a different user per request. For ease, the best approach is to use premium scraping proxies, as these often offer IP rotation from a vast IP pool.
The code below demonstrates how to set up an authenticated premium proxy with Playwright. Replace <PROXY_IP_ADDRESS> and <PROXY_PORT> with the address and port of your proxy server. Also, replace <PROXY_USERNAME> and <PROXY_PASSWORD> with your valid credentials.
// npm install playwright
// npx playwright install
const { chromium } = require("playwright");
(async () => {
// proxy settings
const proxyServer = "http://<PROXY_HOST>:<PROXY_PORT>";
const proxyUsername = "<PROXY_USERNAME>";
const proxyPassword = "<PROXY_PASSWORD>";
// launch browser with proxy
const browser = await chromium.launch({
proxy: {
server: proxyServer,
username: proxyUsername,
password: proxyPassword,
},
});
const page = await browser.newPage();
// open the target page and wait until the network is idle
await page.goto("https://httpbin.io/ip", {
waitUntil: "networkidle",
});
// screenshot the target page
const content = await page.content();
console.log(content);
await browser.close();
})();
Read our detailed tutorial on setting up proxies with Playwright to learn more.
Approach 3: Solve CAPTCHAs
Cloudflare often presents the Turnstile CAPTCHA on an interstitial page or a form field. While this page requires clicking a checkbox to access the target page, Cloudflare performs additional background checks that extend beyond merely clicking a checkbox.
The checkbox is also usually rendered inside a shadow DOM, which makes automated clicking more challenging. Even if the checkbox is clicked programmatically, Cloudflare’s backend verification typically requires a valid CAPTCHA token. So, solving the CAPTCHA rather than just clicking is necessary for successful access.
The most common CAPTCHA-solving service is 2Captcha, a platform that employs human solvers to solve CAPTCHAs in real-time. 2Captcha works by sending the CAPTCHA challenge to a human solver, who then returns the solution as a token.
Let's see how it works with Playwright by solving the CAPTCHA challenge on the 2Captcha Cloudflare Turnstile demo page.
First, install the 2Captcha package:
npm install 2captcha
Cloudflare often presents the Turnstile CAPTCHA on an interstitial page or a form field.
The following script sets up a 2Captcha solver (replace the placeholder with your 2Captcha API key). After opening the target page, it retrieves the CAPTCHA site key from the DOM and passes it to the turnstile solver along with the URL. To solve the CAPTCHA, the response data is then sent to the checkbox field as input:
// npm install 2captcha playwright
// npx playwright install
const { chromium } = require("playwright");
const Captcha = require("2captcha");
// replace with your 2captcha API key
const solver = new Captcha.Solver("<2Captcha_API_KEY>");
(async () => {
// open the target page
const url = "https://2captcha.com/demo/cloudflare-turnstile";
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url);
// extract the sitekey from the data-sitekey attribute
const sitekey = await page.getAttribute(".cf-turnstile", "data-sitekey");
if (!sitekey) {
console.error("Sitekey not found!");
await browser.close();
return;
}
// solve the Turnstile CAPTCHA using 2captcha
const captchaResponse = await solver.turnstile(sitekey, url);
const captchaToken = captchaResponse.data;
// confirm if the token was generated
console.log("Turnstile token:", captchaToken);
// insert the token into the hidden textarea
await page.evaluate((token) => {
const input = document.querySelector(
'textarea[name="cf-turnstile-response"]'
);
if (input) {
input.value = token;
}
}, captchaToken);
// wait to observe the result
await new Promise((resolve) => setTimeout(resolve, 5000));
await browser.close();
})();
Keep in mind that most advanced Cloudflare protections often hide the site key and don't reveal it in the DOM.
Check out our tutorial on bypassing CAPTCHA with Playwright to learn more.
There are two main approaches when it comes to CAPTCHAs: 1) solving them, as we did, and 2) avoiding them, retrying when a request fails. Professional scrapers usually prefer the second method, as it's more reliable and saves you a significant amount of money. An example of a tool that does the latter for you is ZenRows.
If the previous methods didn't work for you, don't panic. You're about to see the lasting solution.
Approach 4: Bypass Advanced Cloudflare Protections With ZenRows
Playwright has strong sides as a browser automation tool, but also multiple limitations when it comes to bypassing Cloudflare's anti-bot measures.
Complementary measures (proxies, stealth plugins, etc.) can partially help, but you may still be blocked if Cloudflare employs more advanced bot detection techniques. Additionally, as Cloudflare security measures evolve, what works today might not work tomorrow.
Therefore, to ensure a more reliable scraping process that gets you the desired data, we recommend using a web scraping solution like the ZenRows Universal Scraper API. ZenRows provides all the toolkits required to scrape any Cloudflare-protected website at scale with zero maintenance requirements. This enables you to focus on core business logic rather than wasting time and resources fixing blocks and broken data pipelines.
ZenRows also has headless browser features for interacting with dynamic pages, making it a suitable alternative to Playwright.
We'll demonstrate the difference by scraping the Cloudflare Challenge page that previously blocked us.
Sign up to open the ZenRows Request Builder. Paste the target URL in the link box, and activate Premium Proxies and JS Rendering.
Choose Node.js as your programming language and select the API connection mode. Copy and paste the generated code into your crawler file:
Here's the generated code:
// npm install axios
const axios = require('axios');
const url = 'https://www.scrapingcourse.com/antibot-challenge';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
url: 'https://api.zenrows.com/v1/',
method: 'GET',
params: {
url: url,
apikey: apikey,
js_render: 'true',
premium_proxy: 'true',
},
})
.then((response) => console.log(response.data))
.catch((error) => console.log(error));
The code outputs the protected website's full-page HTML as shown:
<html lang="en">
<head>
<!-- ... -->
<title>Cloudflare Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Cloudflare challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! 🎉You just bypassed Cloudflare with ZenRows. You now have unlimited access to the data you desire.
Conclusion
You've learned how to bypass Cloudflare while scraping with Playwright. Manual techniques, such as plugins, proxies, and CAPTCHA services, might help to a certain extent. However, they don't work against advanced Cloudflare detection mechanisms.
Meanwhile, gives unwavering success, enabling you to bypass anti-bots and scrape any website at scale without limitations.
Frequent Questions
1. Why Does Cloudflare Keep Blocking Playwright?
Cloudflare continues to block your Playwright scraper due to the automation tool's bot-like properties. Besides, Playwright lacks the built-in mechanisms to evade Cloudflare, making it susceptible to continuous blocking. Even if you add a stealth plugin, it doesn't account for Cloudflare's advanced detection measures.
2. Can Playwright Bypass Cloudflare Turnstile?
Bypassing Cloudflare Turnstile with Playwright is possible, but it's quite challenging. You may need to apply techniques such as integrating a CAPTCHA solver, proxies, a stealth plugin, and sometimes manual evasion tweaks. In most cases, these solutions are ineffective, requiring the use of more reliable, ready-made scraper APIs.
3. Why Should You Replace Playwright With ZenRows for Bypassing Cloudflare?
Playwright's browser instance consumes a significant amount of memory, which reduces scraping performance. ZenRows, on the other hand, provides a lightwright API, despite supporting advanced features like anti-bot auto-bypass, premium proxy rotation, fingerprint spoofing, browser automation, and more. ZenRows guarantees success with optimized scraping performance.