Change the Selenium User Agent: Steps & Best Practices

Sergio Nonide
Sergio Nonide
Updated: January 27, 2025 · 7 min read
Table of contents

Selenium is a popular tool used for web scraping, but it might get you blocked easily. In this tutorial, you'll learn about the role of the User Agent in Selenium plays and how to change it.

What Is the Selenium User Agent

HTTP headers play a vital role in communicating with web servers, and the User-Agent is one of its key elements. It provides information about the browser, operating system, and device you're using to send the request. Let's see an example:

"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"


The above User-Agent reveals that the request originates from a Windows-based Chrome browser.

Unfortunately, relying on the default Selenium User Agent can easily lead to detection and getting blocked by websites.

Does Selenium Have a User Agent

Selenium's default User Agent will depend on the Web Driver you use. The Selenium Web Driver is the key component allowing you to interact with a specific browser. Luckily Web Drivers also allow customizing the User-Agent for scraping purposes.

For example, if you use a Chrome Web Driver with the default options on a Mac computer, your Selenium User Agent could look something like this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36


When doing web scraping, it is common to configure Selenium as a headless browser. In that case, you will get a slightly different User Agent that might look like this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/114.0.5735.106 Safari/537.36


This User Agent can be interpreted by a website like a bot making a request, and in consequence, they can block your scraper. Learning how to change the User Agent in Selenium will help you overcome potential blocks.

How to Change the User Agent in Selenium Web Driver

In this section, you'll learn the step-by-step process of changing the User-Agent in Selenium Web Driver.

How to Set a Custom User-Agent in Selenium

Here's what you need to do in a nutshell:

First, import the necessary libraries and define your custom User-Agent. You can grab one from our list of web scraping User Agents.

program.py 
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
# Define a custom user agent
my_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"

Then, initialize a Chrome Options object and set the custom User-Agent.

program.py 
# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless")

# Set the custom User-Agent
chrome_options.add_argument(f"--user-agent={my_user_agent}")

Next, create a new instance of ChromeDriver with the desired options, and make a request to your target website.

program.py 
# Create a new instance of ChromeDriver with the desired options
driver = webdriver.Chrome(options=chrome_options)

# Make a request to your target website.
driver.get("https://www.seleniumhq.org/download/")

Verify that you've successfully changed the User-Agent by printing it.

program.py 
# Get user Agent with execute_script
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

As a final step, close the driver.

program.py 
# Close the driver
driver.quit()

This is the complete Python code to set a custom User-Agent in Selenium, which will make a request to the target page and will print the new UA that we set.

program.py 
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
 
# Define a custom user agent
my_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless")
 
# Set the custom User-Agent
chrome_options.add_argument(f"--user-agent={my_user_agent}")

# Create a new instance of ChromeDriver with the desired options
driver = webdriver.Chrome(options=chrome_options)

# Make a request to your target website.
driver.get("https://www.seleniumhq.org/download/")

# Get user Agent with execute_script
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

# Close the driver
driver.quit()

To learn more, check out our tutorial on web scraping with Selenium and Python.

Frustrated that your web scrapers are blocked once and again?
ZenRows API handles rotating proxies and headless browsers for you.
Try for FREE

Dynamically Change the User-Agent in Python Selenium

Sometimes, if you use the same User-Agent over and over again, websites can block you. A solution is to rotate through a set of predefined User-Agents by doing the following steps:

Let's start by importing the packages that you'll need.

program.py 
import random
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

Now, define a list of User-Agent strings. You can add as many as you want.

program.py 
# List of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    "Mozilla/5.0 (iPhone14,3; U; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/19A346 Safari/602.1"
    # Add more User-Agent strings as needed
]

Initialize a Chrome Options object and use the --headless parameter for web scraping.

program.py 
chrome_options = Options()
chrome_options.add_argument("--headless")

Next, choose a random User-Agent from the previous list and assign it to the random_user_agent variable.

program.py 
random_user_agent = random.choice(user_agents)

The following step is to add the random_user_agent variable to the Chrome options object as an argument.

program.py 
chrome_options.add_argument(f"--user-agent={random_user_agent}")

Time to create a new instance of the web driver with the custom options, which includes the randomly rotated User-Agent.

program.py 
driver = webdriver.Chrome(options=chrome_options)

With the driver, you can make your requests to the target website:

program.py 
driver.get("https://www.seleniumhq.org/download/")

Get the User-Agent used with the following lines:

program.py 
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

Lastly, close the driver.

program.py 
# Close the driver
driver.quit()

This is the complete Python code to dynamically rotate the Selenium User-Agent:

program.py 
import random
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# List of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    "Mozilla/5.0 (iPhone14,3; U; CPU iPhone OS 15_0 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) Version/10.0 Mobile/19A346 Safari/602.1"
    # Add more User-Agent strings as needed
]

# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument("--headless")

# Choose a random User-Agent from the list
random_user_agent = random.choice(user_agents)
chrome_options.add_argument(f"--user-agent={random_user_agent}")

# Create a new instance of ChromeDriver with the desired options
driver = webdriver.Chrome(options=chrome_options)

driver.get("https://www.seleniumhq.org/download/")

# Get user Agent with execute_script
driver_ua = driver.execute_script("return navigator.userAgent")
print("User agent:")
print(driver_ua)

# Close the driver
driver.quit()

Samples of User-Agents

Let's review other examples of User Agents for Selenium:

Change User Agent OS in Selenium

If you want to change just the OS part of a User-Agent, the first parentheses is where you typically need to perform this change.

Let's start by examining the following User-Agent string:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36


In this example, the OS part is (Windows NT 10.0; Win64; x64). It provides information about the client's operating system, including the OS name (Windows NT), version (10.0), and the system architecture (Win64; x64).

With a bit of string manipulation and using our original User-Agent, you could generate these three new User-Agents:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Mozilla/5.0 (Linux: X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Mozilla/5.0 (Android: Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Change User Agent in Firefox Selenium

In the following example, you'll learn how to change the User Agent using Firefox Selenium. We start by defining a list of user_agents. Again, you can add as many as you want to this list. In this example, we used some common Firefox agents because using the Firefox Web Driver with a Chrome User-Agent is suspicious.

program.py 
import random
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

# List of User-Agent strings
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36",
    # Add more User-Agent strings as needed
]

# Set up Firefox profile
profile = FirefoxProfile()

# Choose a random User-Agent from the list
random_user_agent = random.choice(user_agents)
profile.set_preference("general.useragent.override", random_user_agent)

# Set up Firefox options
firefox_options = Options()
firefox_options.profile = profile

# Create a new instance of Firefox WebDriver with the desired options
driver = webdriver.Firefox(options=firefox_options)

# Your scraping logic goes here
# ...

# Close the driver
driver.quit()

As you can see, it was very similar to Chrome Selenium, with the difference that you needed to create a new FirefoxProfile (profile) for Firefox Selenium, which wasn't necessary for Chrome Selenium. As before, we choose a random User-Agent from the list and set it in the profile with the general.useragent.override preference.

We create a new Firefox Options (firefox_options) object and set its profile property with our custom profile.

Finally, we create a new instance of the web driver with the custom options, which includes the randomly rotated User-Agent.

How To Change User-Agents At Scale

Setting up a good User Agent rotation system isn't as simple as it looks. You need to update your list regularly, check if browser versions make sense with operating systems, and remove old combinations.

Plus, websites don't just look at User Agents. They check many other things about your connection, like how you click, your IP address, your browsing patterns, and more. Even perfect User Agent management won't stop them from detecting Selenium.

The most effective solution is to use a web scraping API like ZenRows. It gives you fresh User Agents that always work, manages your connections, supports JavaScript rendering, auto-bypasses any CAPTCHA, and provides you with everything you need to avoid getting blocked.

Let's see how ZenRows works with a protected website that usually blocks Selenium like the Antibot Challenge page.

Start by signing up for a new account to get to the Request Builder.

building a scraper with zenrows
Click to open the image in full screen

Insert the target URL into the link box, enable JS Rendering, and activate Premium Proxies.

Next, choose Python and then click on the API connection mode. After that, copy the generated code and paste it into your script.

scraper.py 
# pip3 install requests
import requests

url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params, print(response.text)

The generated code uses Python's Requests library as the HTTP client. You can install this library using pip:

Terminal 
pip3 install requests

Run the code, and you'll successfully access the page:

Output 
<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉 You've successfully accessed a protected page without using any complex Selenium setup.

Conclusion

This guide has shown you important things about Selenium User Agents:

  • How User Agents work in Selenium WebDriver.
  • Ways to change User Agents for different browsers.
  • How to rotate between different User Agents.
  • Why User Agent management isn't enough on its own.

Remember that most websites use advanced systems to detect automation. Integrate ZenRows to make sure you extract all the data you need without getting blocked. Try ZenRows for free!

Share
Link copied!

Ready to get started?

Up to 1,000 URLs for free are waiting for you
Try ZenRows for Free