How to Bypass Cloudflare With Selenium

Is your scraper getting blocked by Cloudflare? Although Selenium automates web interactions, its plain version is often detected and blocked by Cloudflare. But there are proven ways to bypass Cloudflare using Selenium with the right setup.

• Method #1: Undetected ChromeDriver.
• Method #2: SeleniumBase.
• Method #3: Solve Cloudflare CAPTCHA.
• Method #4: Web Scraping API to Bypass Cloudflare Every Time.

Let's go!

How Does Cloudflare Detect Selenium?

Cloudflare is a content delivery network (CDN) and cybersecurity provider. It blocks malicious HTTP traffic from moving to the server and performs security checks to mitigate Layer 7 (application layer) DDoS attacks.

Unfortunately, Cloudflare's security system doesn't spare web scrapers. It can detect and block automated browsers like Selenium WebDriver, recognizing them as automated bots. If you've encountered the Selenium Cloudflare verify you are human challenge, you’ve likely been flagged by Cloudflare’s security measures.

Some key methods Cloudflare uses to detect headless browsers like Selenium include:

IP Reputation

Cloudflare maintains a database of IP addresses with known behavior patterns, including suspected bots. It considers historical activities, abnormal request frequency, and other behavioral data to determine an IP's reputation. When Selenium makes requests from an IP address with a poor reputation or abnormal behavior, it is more likely to be flagged and blocked.

HTTP Request Header Analysis

Cloudflare also uses HTTP headers to differentiate between a human user and an automated request. Headers such as User-Agent, Accept-Language, Platform, and others provide clues that can expose automation. By default, Selenium's headers are incomplete or missing relevant parameters, revealing automation signatures that Cloudflare can easily detect.

TLS Fingerprinting

Cloudflare can identify bot activity through TLS (Transport Layer Security) fingerprinting, which analyzes the structure of your browser's SSL/TLS handshake. Information such as the cipher suites, extension order, and other unique values can signal whether a request is from a real user or an automated tool. The details sent by Selenium during its handshake may not match those of genuine browsers, making them easier to identify and block.

CAPTCHAs

Cloudflare often uses Turnstile CAPTCHA to verify whether a visitor is human. This CAPTCHA is a barrier to web scraping as it often requires a complex workaround to proceed. Selenium scripts typically struggle to bypass Turnstile CAPTCHA because it requires more low-level behavioral and traffic pattern recognition. And that's beyond the standard capabilities of automation frameworks like Selenium.

Canvas Fingerprinting

Canvas fingerprinting technique uses the browser to draw hidden images on the canvas to generate a unique fingerprint. Cloudflare leverages the generated fingerprint to differentiate between real users and bots by analyzing subtle variations in how the browser draws the canvas. Selenium often lacks the precise graphical output of a genuine user's browser, which can be a dead giveaway to anti-bot systems.

How to Bypass Cloudflare With Selenium

As mentioned, vanilla Selenium can't bypass Cloudflare since it can't access sites with complex anti-bot services. Let's look at four proven methods for bypassing Cloudflare detection with Selenium.

Method #1: Undetected ChromeDriver

Undetected ChromeDriver is a modified version of the Selenium Chrome WebDriver. It features loophole patches to enable you to access protected sites with Selenium without triggering anti-bot measures. One key advantage of the Undetected ChromeDriver is that it makes Selenium appear more human-like.

While not foolproof, Undetected ChromeDriver increases your chances of bypassing bot detection systems like Cloudflare.

To get started with Undetected ChromeDriver in Python, install it using pip:

pip3 install undetected-chromedriver

Let's use this Cloudflare-protected challenge site as a demo website to show how to use the Undetected ChromeDriver.

Import the library and instantiate it in headless mode with use_subprocess set to False. Then, request the target website, wait for the page to load, and take a screenshot.

# pip3 install undetected-chromedriver
import undetected_chromedriver as uc
from time import sleep

if __name__ == "__main__":

    # instantiate a Chrome browser
    driver = uc.Chrome(
        use_subprocess=False,
        headless=True,
    )

    # visit the target page
    driver.get("https://www.scrapingcourse.com/cloudflare-challenge")

    # wait for the interstitial page to load
    sleep(10)

    # take a screenshot of the current page and save it
    driver.save_screenshot("cloudflare-challenge.png")

    # close the browser
    driver.quit()

After a few trials, the above code grabs the site's home page, demonstrating that Undetected ChromeDriver bypassed Cloudflare:

Although Undetected ChromeDriver bypassed the detection, it doesn't work at scale and may even fail if you rerun the code, as the library still leaks some bot-like attributes. It also struggles with sites that have more advanced Cloudflare security, making it unreliable for many modern websites. 

You can set up a proxy with Undetected ChromeDriver or change your User-Agent to increase the chances of bypassing Cloudflare with Selenium. However, these techniques don't guarantee continuous success, as Cloudflare consistently strengthens its security measures to detect such tweaks.

No worries, this is just the first method. There are more techniques you can apply to bypass Cloudflare with Selenium.

Method #2: SeleniumBase

SeleniumBase is a browser automation and web scraping tool in Python that lets you run Selenium in stealth mode using the Undetected ChromeDriver (UC). SeleniumBase is more efficient than using the bare Undetected ChromeDriver library because it uses advanced browser patches to bypass anti-bot checks. Many developers use SeleniumBase Cloudflare techniques to improve their scraping success rate.

Let's see how SeleniumBase performs against the same Cloudflare challenge site. First, install the library using pip:

pip3 install seleniumbase

Start the browser instance in the GUI mode (non-headless) using UC. Then, navigate to the target page with a 6-second reconnection time to allow SeleniumBase to bypass the initial JavaScript challenge.

# pip3 install seleniumbase
from seleniumbase import Driver

# initialize the driver in GUI mode with UC enabled
driver = Driver(uc=True, headless=False)

# set the target URL
url = "https://www.scrapingcourse.com/cloudflare-challenge"

# open URL with a 6-second reconnect time to bypass the initial JS challenge
driver.uc_open_with_reconnect(url, reconnect_time=6)

Use the uc_gui_click_captcha method to click the CAPTCHA checkbox and resolve it if it appears. Note that the uc_gui_click_captcha method only works in the GUI mode (non-headless). Finally, grab a screenshot of the web page:

# ...

# attempt to click the CAPTCHA checkbox if present
driver.uc_gui_click_captcha()

# take a screenshot of the current page and save it
driver.save_screenshot("cloudflare-challenge.png")

# close the browser and end the session
driver.quit()

Combine the snippets, and you'll get the following complete code:

# pip3 install seleniumbase
from seleniumbase import Driver

# initialize the driver in GUI mode with UC enabled
driver = Driver(uc=True, headless=False)

# set the target URL
url = "https://www.scrapingcourse.com/cloudflare-challenge"

# open URL using UC mode with 6 second reconnect time to bypass initial detection
driver.uc_open_with_reconnect(url, reconnect_time=6)

# attempt to click the CAPTCHA checkbox if present
driver.uc_gui_click_captcha()

# take a screenshot of the current page and save it
driver.save_screenshot("cloudflare-challenge.png")

# close the browser and end the session
driver.quit()

You should see the following screenshot upon successful execution:

SeleniumBase bypassed the anti-bot challenge! 

However, it's essential to note that SeleniumBase has its limitations. While it may record a higher success rate than the main Undetected ChromeDriver library, it also doesn't guarantee success at scale. The fact that it only solves Cloudflare CAPTCHA in GUI mode means it uses more memory. This makes it unsuitable for large-scale scraping. 

You can also improve stealth by setting up a proxy with SeleniumBase, but it can still get detected, particularly in headless mode. Additionally, since the library is open-source, Cloudflare's developers can study its mechanisms and adapt their defenses to block it.

Method #3: Solve Cloudflare CAPTCHA

The Cloudflare Turnstile CAPTCHA can appear in-page on web elements, such as form fields, checkout buttons, and more. This CAPTCHA challenge requires more than just clicking a checkbox, as Cloudflare analyzes several bot determinants under the hood that typically require a solution token. 

Even if you automate the checkbox clicking with Selenium, you'll still get blocked without a clearance token. An approach to this challenge is to solve the Turnstile CAPTCHA with a CAPTCHA-solving tool like 2Captcha.

2Captcha employs humans to solve CAPTCHA by sending the CAPTCHA challenge to them. These human solvers then return the solution as a token. Let's see how 2Captcha works with Selenium by solving the CAPTCHA challenge on the 2Captcha Cloudflare Turnstile demo page.

To use 2Captcha with Selenium in Python, install it using pip:

pip3 install twocaptcha

Import twocaptcha with other relevant libraries. Then, set up 2Captcha with your 2Captcha API key:

# pip3 install twocaptcha selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from twocaptcha import TwoCaptcha

# set up 2Captcha solver
solver = TwoCaptcha("<2Captcha_API_KEY>")

# set up the WebDriver in headless mode
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)

Open the target site and extract the Turnstile data site key dynamically. Initiate CAPTCHA solving and obtain the solution token. Inject this token into the required field to solve the CAPTCHA and access the blocked page. The following code checks if CAPTCHA is solved by clicking the provided confirmation button after injection:

# ...

# open the target site
url = "https://2captcha.com/demo/cloudflare-turnstile"
driver.get(url)

# wait for the page to load
time.sleep(5)

# extract the sitekey
sitekey_elem = driver.find_element(By.CSS_SELECTOR, ".cf-turnstile")
sitekey = sitekey_elem.get_attribute("data-sitekey")

# solve the captcha
result = solver.turnstile(sitekey=sitekey, url=url)
captcha_token = result["code"]

# wait for the page to be ready for interaction
time.sleep(2)

# insert the token into the textarea
textarea = driver.execute_script(
    "return document.querySelector('input[name=\"cf-turnstile-response\"]');"
)
driver.execute_script("arguments[0].value = arguments[1];", textarea, captcha_token)

# wait for the token to be set
time.sleep(2)

# get the check button element
check_button = driver.find_element(By.CSS_SELECTOR, 'button[data-action="demo_action"]')

# scroll to the button to ensure the check button is in view
driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", check_button)
time.sleep(0.5)

# click the "Check" button
check_button.click()

# wait for either success or failure message
wait = WebDriverWait(driver, 10)
try:
    success = wait.until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "p._successMessage_1ndnh_1"))
    )
    print("Success Message:", success.text)
except Exception:
    try:
        error = WebDriverWait(driver, 3).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "div._alertBody_bl73y_16"))
        )
        print("Error Message:", error.text)
    except Exception:
        print("Neither success nor error message appeared.")

driver.quit()

This is the complete code:

# pip3 install twocaptcha selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from twocaptcha import TwoCaptcha

# set up 2Captcha solver
solver = TwoCaptcha("<2Captcha_API_KEY>")

# set up the WebDriver in headless mode
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)


# open the target site
url = "https://2captcha.com/demo/cloudflare-turnstile"
driver.get(url)

# wait for the page to load
time.sleep(5)

# extract the sitekey
sitekey_elem = driver.find_element(By.CSS_SELECTOR, ".cf-turnstile")
sitekey = sitekey_elem.get_attribute("data-sitekey")

# solve the captcha
result = solver.turnstile(sitekey=sitekey, url=url)
captcha_token = result["code"]

# wait for the page to be ready for interaction
time.sleep(2)

# insert the token into the textarea
textarea = driver.execute_script(
    "return document.querySelector('input[name=\"cf-turnstile-response\"]');"
)
driver.execute_script("arguments[0].value = arguments[1];", textarea, captcha_token)

# wait for the token to be set
time.sleep(2)

# get the check button element
check_button = driver.find_element(By.CSS_SELECTOR, 'button[data-action="demo_action"]')

# scroll to the button to ensure the check button is in view
driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", check_button)
time.sleep(0.5)

# click the "Check" button
check_button.click()

# wait for either success or failure message
wait = WebDriverWait(driver, 10)
try:
    success = wait.until(
        EC.presence_of_element_located((By.CSS_SELECTOR, "p._successMessage_1ndnh_1"))
    )
    print(success.text)
except Exception:
    try:
        error = WebDriverWait(driver, 3).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "div._alertBody_bl73y_16"))
        )
        print(error.text)
    except Exception:
        print("Neither success nor error message appeared.")

driver.quit()

Run the above code, and it should trigger a success message, proving that Turnstile CAPTCHA has been solved:

Captcha is passed successfully!

Good job solving that CAPTCHA! However, this method is still unreliable for handling advanced Cloudflare protection. Most modern sites often hide the data site key, so you can't extract it from the DOM.

Check out our comprehensive guide on bypassing CAPTCHA with Selenium to learn more.

That said, the next solution works consistently and is the most recommended way to bypass Cloudflare at scale.

Method #4: Web Scraping API to Bypass Cloudflare Every Time

For guaranteed success in bypassing Cloudflare, one tried-and-true method is to use a web scraping API like the ZenRows Universal Scraper API.

ZenRows provides all the tools you need to scrape any Cloudflare-protected website at scale without limitations. ZenRows provides an auto-managed and auto-scaled consistent access to your desired data. 

With ZenRows, you only need to make a single API call with your chosen programming language and watch it bypass the anti-bot behind the scenes. This way, you can focus on core business logic rather than waste time fixing blocks and broken data pipelines. ZenRows also serves as a headless browser, making it a suitable replacement for Selenium. It features various JavaScript instructions for interacting with web pages like humans.

To see how ZenRows works, let's use it to access and scrape the previous Cloudflare challenge page.

Sign up and open the ZenRows Request Builder. Paste the target URL in the link box and activate Premium Proxies and JS Rendering.

Select Python as your programming language and choose the API connection mode. Copy and paste the generated code into your Python script.

The generated code should look like this:

# pip install requests
import requests

url = 'https://www.scrapingcourse.com/cloudflare-challenge'
apikey = '<YOUR_ZENROWS_API_KEY>'

params = {
    'url': url,
    'apikey': apikey,
    'js_render': 'true',
    'premium_proxy': 'true',
}

response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

The above scraper accesses the protected website and scrapes its full-page HTML, as shown:

<html lang="en">
<head>
    <!-- ... -->
    <title>Cloudflare Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Cloudflare challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! Your scraper now bypasses the highest Cloudflare protection with ZenRows.

Conclusion

In this Selenium Cloudflare bypass guide, you've learned four ways to improve vanilla Selenium and avoid Cloudflare detection. Open-source solutions such as Undetected ChromeDriver and SeleniumBase, as well as even setting them up with proxies, offer varying levels of success and are unreliable at scale.

However, the only method that always works is employing a web scraping API, such as ZenRows. This solution handles all anti-bot bypass technicalities behind the scenes while you focus on your scraping logic.

Try ZenRows for free without a credit card!