How to Change User Agent to Bypass Cloudflare

Aitor Mato
Aitor Mato
Updated: January 28, 2025 · 4 min read
Table of contents

Getting blocked by Cloudflare? Changing your User-Agent to mimic real browser traffic can help your scraper slip through undetected and access protected pages.

What Is the User Agent (and Why Cloudflare May Block Yours)?

HTTP request headers are a set of key-value pairs sent by the client to the server that provide essential information about the request. One of the key elements of these headers is the User Agent (UA).

The User Agent string helps identify the client making the request. The User Agent of most web scraper bots significantly differs from the User Agents of regular browsers, which is why Cloudflare detects and blocks them.

For example, this is what a Chrome User Agent looks like:

Example 
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.127 Safari/537.36

In contrast, let's see Selenium's User Agent. The following code targets HTTPBin, a page that returns the User Agent of the client making the request:

scraper.py 
# import the required libraries
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

# run Chrome in headless mode
options = Options()
options.add_argument("--headless=new")

# start a driver instance
driver = webdriver.Chrome(options=options)

# open the target website
driver.get("https://httpbin.io/user-agent")

# print the HTML
print(driver.find_element(By.TAG_NAME, "body").text)

# release the allocated resources
driver.quit()

You'll get the following output on running this code:

Output 
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/126.0.0.0 Safari/537.36"
}

In this output, the User Agent string indicates that the browser is running in headless mode, as shown by HeadlessChrome/126.0.0.0. This is how Cloudflare detects and blocks such requests.

Let's learn how to set up the User Agent to lower the chances of Cloudflare detection.

Frustrated that your web scrapers are blocked once and again?
ZenRows API handles rotating proxies and headless browsers for you.
Try for FREE

Change Your User Agent to Avoid Cloudflare Detection

Changing your User Agent is a simple yet effective way to avoid Cloudflare detection. In this section, we'll walk you through the process using Python and Selenium as an example.

Set Custom User Agent

First, grab the latest User Agent from our list of top User Agents for web scraping.

Next, import the necessary libraries in your code.

scraper.py 
# pip3 install selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

Initialize a Chrome Options object and configure Selenium to use a custom User Agent.

scraper.py 
# ...

# create a Chrome Options instance
options = Options()
options.add_argument("--headless=new")

# set a custom User Agent
custom_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
options.add_argument(f"user-agent={custom_user_agent}")

# Start the WebDriver instance with the options
driver = webdriver.Chrome(options=options)

Open the HTTPBin target website to verify the User Agent has been set correctly. Finally, print the response HTML and close the driver.

scraper.py 
# ...

# open the target website
driver.get("https://httpbin.io/user-agent")

# print the User Agent to verify
print(driver.find_element(By.TAG_NAME, "body").text)

# release the allocated resources
driver.quit()

You'll get the following output:

Output 
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
}

Congrats! You successfully modified the User Agent.

While using a single custom User Agent might work for small-scale scraping, more is needed for extracting data from Cloudflare-protected pages at scale. Repeated requests from the same User Agent can still be detected and blocked by Cloudflare and other WAF services like PerimeterX, DataDome, Imperva/Incapsula, etc.

Let's learn how to build a manual User Agent rotator to reduce the chances of detection further.

Rotate Your User Agent

Rotating User Agents is critical to avoid getting blocked, as too many requests from the same User Agent can easily trigger detection mechanisms. Here's how you can rotate your User Agents randomly.

Grab a few User Agents from our list of top User Agents for web scraping.

Example 
# list of User Agent strings
user_agents = [
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.4; rv:124.0) Gecko/20100101 Firefox/124.0",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
]

Use Python's random.choice method to choose a User Agent from this list randomly.

Let's modify the previous code and implement the rotating User Agent functionality:

scraper.py 
import random
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

# list of User Agent strings
user_agents = [
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14.4; rv:124.0) Gecko/20100101 Firefox/124.0",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36",
]

# create a Chrome Options instance
options = Options()
options.add_argument("--headless=new")

# set a random User Agent
random_user_agent = random.choice(user_agents)
options.add_argument(f"user-agent={random_user_agent}")

# Start the WebDriver instance with the options
driver = webdriver.Chrome(options=options)

# open the target website
driver.get("https://httpbin.io/user-agent")

# print the User Agent to verify
print(driver.find_element(By.TAG_NAME, "body").text)

# release the allocated resources
driver.quit()

By running this code, the output would show the randomly selected User Agent, such as:

Output 
# request 1
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
}

# request 2
{
  "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
}

# ...

Constructing a proper User Agent string is crucial to avoid detection. Many anti-bot systems, such as Cloudflare, look for inconsistencies in these details. For instance, using WebKit versions associated with Safari in a modern Chrome User Agent string is a clear mismatch.

Inconsistent User Agent Example

Example 
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/605.1.15

In this incorrect example, AppleWebKit/605.1.15 is associated with Safari, not Chrome. AppleWebKit/605.1.15 and Safari/605.1.15 indicate a Safari browser version, but the string claims to be Chrome (Chrome/126.0.0.0). This inconsistency signals that the User Agent string is fabricated.

Correct User Agent Example

Example 
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36

In this correct example, AppleWebKit/537.36 is used by both Chrome and Safari. It's paired with the Chrome version Chrome/91.0.4472.124 and Safari/537.36.

Keeping your User Agents up-to-date is crucial to avoid detection. Outdated User Agents can be easily flagged by security systems. Regularly refresh your list by sourcing the latest User Agents from reliable repositories or websites, and use automation tools to fetch them programmatically.

Ensure your list includes various browsers, versions, platforms, and devices to mimic real user traffic accurately.

However, maintaining this can be challenging and time-consuming. In the next section, we'll explore a better alternative to bypass Cloudflare efficiently.

How to Bypass Cloudflare Every Time

Creating a reliable User Agent rotation system requires constant work. You need to keep updating browser versions, ensure consistency between operating systems and browser details, and remove outdated combinations.

Plus, Cloudflare looks beyond User Agents to detect bots. It analyzes your browsing patterns, request behavior, TLS fingerprints, and more. Even with perfect User Agent management, you might still trigger Cloudflare's protection.

A better solution is using ZenRows' Universal Scraper API, it provides auto-rotating up-to-date User Agents, premium proxy, JavaScript rendering, CAPTCHA auto-bypass, and everything you need to avoid getting blocked.

Let's see how ZenRows performs against a protected page like the Antibot Challenge page.

Start by signing up for a new account, and you'll get to the Request Builder.

building a scraper with zenrows
Click to open the image in full screen

Paste the target URL, enable JS Rendering, and activate Premium Proxies.

Next, select Python and click on the API connection mode. Then, copy the generated code and paste it into your script.

scraper.py 
# pip3 install requests
import requests

url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params, print(response.text)

The generated code uses Python's Requests library as the HTTP client. You can install this library using pip:

Terminal 
pip3 install requests

Run the code, and you'll successfully access the page:

Output 
<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉 You've accessed a Cloudflare-protected page without any complex User Agent configuration.

Conclusion

This guide has shown you important things about bypassing Cloudflare with User Agents:

  • How Cloudflare uses User Agents for detection.
  • Ways to set proper User Agents in your requests.
  • How to rotate between different User Agents.
  • Why User Agent management alone isn't enough.

Remember that Cloudflare uses many methods to detect bots. Instead of managing everything yourself, use ZenRows to make sure you extract all the data you need without getting blocked. Try ZenRows for free!

Share
Link copied!

Ready to get started?

Up to 1,000 URLs for free are waiting for you
Try ZenRows for Free