Is Cloudflare blocking your scraper with a 403 Forbidden error? You're being detected as a bot, and that needs to stop!
We'll show you four ways to bypass the Cloudflare 403 Forbidden error and scrape without getting blocked.
But first, let's understand what the error really means.
What Is Error 403?
An error 403 means that the server understands your request but forbids it. That can happen when scraping with Python, NodeJS, cURL, or any other programming language.
The 403 Forbidden error usually comes up when Cloudflare detects bot-like signals, such as unusual traffic from the same IP, missing fingerprints, suspicious user interactions, such as rapidly filling out a form, etc.
After detecting these bot signals, Cloudflare's firewall assumes you're a threat and sometimes displays an Error 1020 or an interstitial challenge screen like the one below, representing the 403 status code.
Here are four methods you can use to bypass the 403 Forbidden error without getting blocked.
1. Use a Web Scraping API
The best way to bypass anti-bots during scraping is to use a web scraping API.
Web scraping APIs like ZenRows allow you to scrape protected web pages without triggering the Cloudflare 403 forbidden error. As an all-in-one web scraping solution, ZenRows handles all the bypass logic behind the scenes with a single API call, so you don't have to deal with the complexities manually.
Let's try using ZenRows to scrape the Cloudflare Challenge page, a Cloudflare-protected webpage.
All you need to get started is sign up to open the ZenRows Request Builder. Once in the Request Builder, enter the target site's URL in the link box and activate Premium Proxies and JS Rendering. Select your programming language (Python, in this case) and choose the API connection mode.
Copy and paste the generated code into your scraper file.
Since we've used Python in this example, ensure you install the Requests library if you've not done so already:
pip3 install requests
The generated Python code should look like the following:
# pip3 install requests
import requests
url = "https://www.scrapingcourse.com/cloudflare-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
The above code bypasses the Cloudflare 403 Forbidden error and scrapes the target's full-page HTML:
<html lang="en">
<head>
<!-- ... -->
<title>Cloudflare Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Cloudflare challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congrats 🎉! You've bypassed Cloudflare with Python and Zenrows!
While this is the recommended approach to bypass Cloudflare error 403, there are other methods worth trying if you prefer tweaking evasion strategies manually.
2. Get Premium Proxies
Some websites only show the Cloudflare 403 forbidden error when you exceed the rate limit or use a disreputed IP address.
Web scraping proxies act as middlemen between your scraper and the target server, allowing you to route your requests through different IP addresses. You can use free or premium proxies. However, while free proxies are readily available, they often pose challenges, such as slow speeds, high failure rates, and a high risk of being banned.
Most web scraping APIs embed premium proxies as part of their services and help you auto-rotate them under the hood. However, using proxies alone is often more cost-effective, especially when the target site only shows the Cloudflare 403 forbidden error when you exceed the rate limit, hit geo-restriction, or use a low-quality IP address.
Premium residential proxies are the most reliable for scraping. They route requests through IP addresses assigned to real users by network providers. Most premium proxy providers also offer IP rotation to distribute traffic across several residential locations.
These attributes make your request appear natural and increase its chances of bypassing the Cloudflare 403 forbidden error.
ZenRows is one of the top residential proxy providers. It rotates proxies from a pool of 55 million residential IPs distributed across 185+ countries. ZenRows's residential proxy also features flexible geotargeting, allowing you to access geo-restricted content.
Let's see how ZenRows' residential proxies work by requesting <https://httpbin.io/ip> with Python's Requests.
Sign up to open the Request Builder. Then, head over to Residental Proxies. Copy the proxy URL containing your proxy credentials (password and username).
Implement the copied proxy credentials as shown below:
# pip3 install requests
import requests
# define the proxy with your authentication credentials
proxies = {
"http": "http://<ZENROWS_PROXY_USERNAME>:<ZENROWS_PROXY_PASSWORD>@superproxy.zenrows.com:1337",
"https": "https://<ZENROWS_PROXY_USERNAME>:<ZENROWS_PROXY_PASSWORD>@superproxy.zenrows.com:1338",
}
# set the target URL
url = "https://httpbin.io/ip"
# request the target website
response = requests.get(url, proxies=proxies)
# print the response to see if the proxy worked
print(response.text)
The above rotates your IP per request. Here's a sample output for three consecutive requests:
# request 1
{
"origin": "78.10.89.181:58974"
}
# request 2
{
"origin": "24.115.78.114:38732"
}
# request 3
{
"origin": "81.190.33.78:33908"
}
Check out how to use a proxy with Python's Requests to learn more.
However, proxies are insufficient for bypassing advanced anti-bot protections that flag bot-like attributes beyond IP reputation. If your request still gets blocked after trying premium proxies, use a web scraping API, as recommended in the previous section.
3. Bypass Cloudflare 403 Forbidden with a Stealth Browser
Cloudflare uses security techniques, including TLS fingerprinting, browser fingerprinting, behavioral trends, and more, to identify and block web clients. For instance, during a TLS handshake, Cloudflare analyzes multiple parameters to determine if the incoming request is legitimate or potentially malicious.
Regular headless browsers like Playwright, Selenium, and Puppeteer can't bypass these detection mechanisms because they leak bot signals, including WebDriver, faulty WebGL fingerprints, misconfigured browser runtime, and more.
Stealth headless browsers address the vulnerabilities in these base headless browsers by patching them with evasion techniques, increasing the likelihood of bypassing the Cloudflare 403 error.
Popular open-source stealth libraries include:
- Puppeteer Extra Stealth for Puppeteer.
- Playwright stealth and Camoufox for Playwright.
- SeleniumBase with Undetected ChromeDriver for Selenium.
Stealth plugins, including Playwright Stealth and Puppeteer Stealth, were effective against anti-bots when they were first released. Unfortunately, they're ineffective now because they can't keep up with anti-bot updates.
Camoufox and SeleniumBase are relatively new and still work to some extent on a small scale.
To see how these stealth tools work, let's attempt to bypass the Cloudflare 403 Forbidden error using SeleniumBase by scraping this Cloudflare Challenge page.
To begin, install SeleniumBase using pip:
pip3 install seleniumbase
Next, import the required libraries and set up a non-headless (GUI) SeleniumBase driver in Undetected ChromeDriver (uc=True) mode. Open the target site and execute the Cloudflare Turnstile click using SeleniumBase's uc_gui_click_captcha to solve the Cloudflare CAPTCHA. Note that we've used the GUI mode because SeleniumBase only supports Cloudflare Turnstile bypass in the non-headless mode:
Here's the full code:
# pip3 install seleniumbase
from seleniumbase import Driver
import time
# create a new Driver instance in UC mode
driver = Driver(uc=True)
# open the target site
driver.get("https://www.scrapingcourse.com/cloudflare-challenge/")
# implicitly wait for the DOM to load
driver.implicitly_wait(10)
# attempt to click the CAPTCHA checkbox
driver.uc_gui_click_captcha()
# wait for the challenge to complete
time.sleep(10)
# print the page content after passing the challenge
print(driver.get_page_source())
driver.quit()
If all looks good, the above code should open the browser in GUI mode, click the Cloudflare checkbox, and output the website content, as shown:
<html lang="en">
<head>
<!-- ... -->
<title>Cloudflare Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Cloudflare challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
While an open-source stealth tool like SeleniumBase or Camoufox might help you bypass initial Cloudflare 403 blocks, they don't guarantee sustainable success.
For instance, one limitation of SeleniumBase is that its Cloudflare evasion mode only works in GUI mode, which consumes a lot of system memory. So, it's unsuitable for large-scale scraping. Additionally, like many other open-source tools, Cloudflare will eventually also clamp down on its evasion techniques, rendering it ineffective for scraping protected pages.
If you want guaranteed success at bypassing Cloudflare 403 at scale, the best option remains using a dedicated scraper API, such as ZenRows.
Conclusion
You've learned three solid ways to bypass the Cloudflare 403 error. While solutions like premium proxies and open-source stealth browsers can increase evasion chances, they don't guarantee continuous success. Open-source stealth browsers are also unscalable and have high memory overhead.
The easiest way to bypass the Cloudflare 403 error is to use ZenRows, an all-in-one web scraping solution at scale. Whether scraping with an HTTP client or a headless browser, ZenRows provides an auto-scaled, auto-managed infrastructure that suits your needs.
Try ZenRows for free now or speak with sales!
Frequent Questions
What Does 403 Forbidden Mean?
A 403 Forbidden error indicates that the server understands your request but refuses to authorize it. This often happens when Cloudflare detects bot-like activity, such as unusual traffic patterns, missing browser fingerprints, or suspicious interactions, and blocks access to protect the website.
Why Do I Get a 403 Forbidden Message?
You may get a 403 Forbidden error due to:
- Sending multiple requests from the same IP in a short time.
- Using incomplete or bot-like HTTP headers.
- Attempting to access restricted pages or violating the website's terms.
- Being flagged by Cloudflare's firewall for suspicious activity.
How to Fix 403 Forbidden Cloudflare?
To fix the Cloudflare 403 Forbidden error:
- Use ZenRows: ZenRows Universal Scraper API that handles proxies, headers, and anti-bot measures automatically.
- Get Premium Proxies: Rotate IPs to avoid detection and rate limits.
- Use Headless Browsers: Simulate human interactions with tools like Puppeteer or Playwright.
- Fortify Browsers: Use stealth plugins or advanced solutions like the ZenRows Scraping Browser for better evasion.
Why Is My IP Blocked by Cloudflare?
Cloudflare blocks IPs for reasons such as:
- Sending excessive or rapid requests.
- Using flagged or disreputable IPs, often from free proxies.
- Failing to meet browser fingerprinting checks.
- Violating access rules, such as attempting to scrape geo-restricted content.
How do I stop Cloudflare from blocking me?
To stop Cloudflare from blocking you during web scraping, you can use premium proxies or stealth browsers to bypass its initial anti-bot checks.
If Cloudflare blocks you only when you exceed a specific number of requests or when accessing localized content in another region, a premium proxy service with geolocation and IP auto-rotation features is usually enough. Open-source stealth browsers work for basic, small-scale scraping.
However, the most reliable way to stop Cloudflare from blocking you is to use a dedicated web scraping solution, such as ZenRows. This automatically handles anti-bot bypass under the hood, enabling you to focus on core data tasks rather than wasting time and resources debugging failed scrapers.