Change the Selenium User Agent: Steps & Best Practices

Selenium is a popular tool used for web scraping, but it might get you blocked easily. In this tutorial, you'll learn about the role of the User Agent in Selenium plays and how to change it.

What Is the Selenium User Agent

HTTP headers play a vital role in communicating with web servers, and the User-Agent is one of its key elements. It provides information about the browser, operating system, and device you're using to send the request. Let's see an example:

"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"

The above User-Agent reveals that the request originates from a Windows-based Chrome browser.

Unfortunately, relying on the default Selenium User Agent can easily lead to detection and getting blocked by websites.

Does Selenium Have a User Agent

Selenium's default User Agent will depend on the Web Driver you use. The Selenium Web Driver is the key component allowing you to interact with a specific browser. Luckily Web Drivers also allow customizing the User-Agent for scraping purposes.

For example, if you use a Chrome Web Driver with the default options on a Mac computer, your Selenium User Agent could look something like this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36

When doing web scraping, it is common to configure Selenium as a headless browser. In that case, you will get a slightly different User Agent that might look like this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/114.0.5735.106 Safari/537.36

This User Agent can be interpreted by a website like a bot making a request, and in consequence, they can block your scraper. Learning how to change the User Agent in Selenium will help you overcome potential blocks.

How to Change the User Agent in Selenium Web Driver

In this section, you'll learn the step-by-step process of changing the User-Agent in Selenium Web Driver.

How to Set a Custom User-Agent in Selenium

Here's what you need to do in a nutshell:

First, import the necessary libraries and define your custom User-Agent. You can grab one from our list of web scraping User Agents.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
# Define a custom user agent
my_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"

Then, initialize a Chrome Options object and set the custom User-Agent.

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless")

# Set the custom User-Agent
chrome_options.add_argument(f"--user-agent={my_user_agent}")

Next, create a new instance of ChromeDriver with the desired options, and make a request to your target website.

# Create a new instance of ChromeDriver with the desired options
driver = webdriver.Chrome(options=chrome_options)

# Make a request to your target website.
driver.get("https://www.seleniumhq.org/download/")

Verify that you've successfully changed the User-Agent by printing it.

# Get user Agent with execute_script
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

As a final step, close the driver.

# Close the driver
driver.quit()

This is the complete Python code to set a custom User-Agent in Selenium, which will make a request to the target page and will print the new UA that we set.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
# Define a custom user agent
my_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless")
 
# Set the custom User-Agent
chrome_options.add_argument(f"--user-agent={my_user_agent}")

# Create a new instance of ChromeDriver with the desired options
driver = webdriver.Chrome(options=chrome_options)

# Make a request to your target website.
driver.get("https://www.seleniumhq.org/download/")

# Get user Agent with execute_script
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

# Close the driver
driver.quit()

To learn more, check out our tutorial on web scraping with Selenium and Python.

Dynamically Change the User-Agent in Python Selenium

Sometimes, if you use the same User-Agent over and over again, websites can block you. A solution is to rotate through a set of predefined User-Agents by doing the following steps:

Let's start by importing the packages that you'll need.

import random
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

Now, define a list of User-Agent strings. You can add as many as you want.

# List of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    "Mozilla/5.0 (iPhone14,3; U; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/19A346 Safari/602.1"
    # Add more User-Agent strings as needed
]

Initialize a Chrome Options object and use the --headless parameter for web scraping.

chrome_options = Options()
chrome_options.add_argument("--headless")

Next, choose a random User-Agent from the previous list and assign it to the random_user_agent variable.

random_user_agent = random.choice(user_agents)

The following step is to add the random_user_agent variable to the Chrome options object as an argument.

chrome_options.add_argument(f"--user-agent={random_user_agent}")

Time to create a new instance of the web driver with the custom options, which includes the randomly rotated User-Agent.

driver = webdriver.Chrome(options=chrome_options)

With the driver, you can make your requests to the target website:

driver.get("https://www.seleniumhq.org/download/")

Get the User-Agent used with the following lines:

driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

Lastly, close the driver.

# Close the driver
driver.quit()

This is the complete Python code to dynamically rotate the Selenium User-Agent:

import random
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# List of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    "Mozilla/5.0 (iPhone14,3; U; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/19A346 Safari/602.1"
    # Add more User-Agent strings as needed
]

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless")

# Choose a random User-Agent from the list
random_user_agent = random.choice(user_agents)
chrome_options.add_argument(f"--user-agent={random_user_agent}")

# Create a new instance of ChromeDriver with the desired options
driver = webdriver.Chrome(options=chrome_options)

driver.get("https://www.seleniumhq.org/download/")

# Get user Agent with execute_script
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

# Close the driver
driver.quit()

Samples of User-Agents

Let's review other examples of User Agents for Selenium:

Change User Agent OS in Selenium

If you want to change just the OS part of a User-Agent, the first parentheses is where you typically need to perform this change.

Let's start by examining the following User-Agent string:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

In this example, the OS part is (Windows NT 10.0; Win64; x64). It provides information about the client's operating system, including the OS name (Windows NT), version (10.0), and the system architecture (Win64; x64).

With a bit of string manipulation and using our original User-Agent, you could generate these three new User-Agents:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Mozilla/5.0 (Linux: X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Mozilla/5.0 (Android: Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Change User Agent in Firefox Selenium

In the following example, you'll learn how to change the User Agent using Firefox Selenium. We start by defining a list of user_agents. Again, you can add as many as you want to this list. In this example, we used some common Firefox agents because using the Firefox Web Driver with a Chrome User-Agent is suspicious.

import random
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

# List of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    # Add more User-Agent strings as needed
]

# Set up Firefox profile
profile = FirefoxProfile()

# Choose a random User-Agent from the list
random_user_agent = random.choice(user_agents)
profile.set_preference("general.useragent.override", random_user_agent)

# Set up Firefox options
firefox_options = Options()
firefox_options.profile = profile

# Create a new instance of Firefox WebDriver with the desired options
driver = webdriver.Firefox(options=firefox_options)

# Your scraping logic goes here
# ...

# Close the driver
driver.quit()

As you can see, it was very similar to Chrome Selenium, with the difference that you needed to create a new FirefoxProfile (profile) for Firefox Selenium, which wasn't necessary for Chrome Selenium. As before, we choose a random User-Agent from the list and set it in the profile with the general.useragent.override preference.

We create a new Firefox Options (firefox_options) object and set its profile property with our custom profile.

Finally, we create a new instance of the web driver with the custom options, which includes the randomly rotated User-Agent.

How To Change the User-Agent At Scale

When it comes to web scraping in practice, more than one User-Agent is needed. The required step is to use a pool of many of User-Agents and rotate them randomly to increase your chances of avoiding detection. 

Soon, you'll find that managing User-Agents isn't only a matter of quantity but also quality. You need to ensure each User-Agent name, version, and operating system matches the correct driver and also makes sense to avoid triggering anti-bot protection systems.

Examples of suspicious User-Agents are the ones of outdated browsers or those whose Operating System and web browser combination don't match.

Besides the User-Agent, websites may check other header strings. In these cases, you'll need to simulate a real browser by setting more headers, like Referer, Accept-Language, Accept-Encoding, Connection, and Cookie. If you want to learn more about this, check our guide on the most common HTTP headers for web scraping.

However, even after setting all these headers, you can get blocked because modern web security systems use diverse techniques to detect scrapers and other bots. If you want to learn more, check the tried-and-tested anti-scraping techniques.

On the bright side, there is a solution that implements all those techniques and lets you scrape quickly and easily. ZenRows web scraping API has proven invisible to all anti-bot systems, and it works with any programming language.

With ZenRows, you don't have to worry about configuring and maintaining Selenium. If you want to use a headless browser, all you have to do is to activate the JavaScript Rendering parameter. Also, ZenRows sets and auto-rotates the User-Agent for you by default.

Starting with ZenRows, it's straightforward. Sign up to get your free API key, and you'll get to the Request Builder page. There, you'll get the code you need to use.

Conclusion

Setting and randomizing User-Agents is essential in web scraping to bypass anti-bot systems. Using different and well-formed User-Agent strings for each request increases the chances of avoiding detection and blocks by the target website.

Unfortunately, that may not be enough when web scraping at scale, so you'll need to use other advanced techniques. To learn more, check our guide on web scraping without getting blocked.

To make the whole process easier, you can rely on ZenRows. It can integrate seamlessly with your current Python and Selenium implementation, so you don't need to start from scratch. Use the 1,000 free API credits to test it for your next scraping project.

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn. It looks like this tutorial has helped you. Be someone's hero today: Share on Twitter Share on LinkedIn