Undetected ChromeDriver in Python Selenium & Common Errors

Selenium comes with a default WebDriver that often fails to bypass scraping anti-bots. Yet you can complement it with Undetected ChromeDriver, a third-party WebDriver tool that will do a better job.

In this tutorial, you'll learn how to use Undetected ChromeDriver with Selenium in Python and solve the most common errors.

What Is Undetected ChromeDriver?

Undetected ChromeDriver is a Selenium WebDriver optimized to avoid triggering anti-bots. 

Some examples? Cloudflare and Akamai. It works with Google Chrome, Brave and many other Chromium-based browsers.

How to Use Undetected ChromeDriver in Python with Selenium?

Let's see what you'll need to get started!

Prerequisites

Let's see what you'll need to get started!

To use Undetected ChromeDriver 2, you'll need the following:

• Selenium because it's the base.
• Python 3 since the driver works only with Python 3.6 or higher.
• Chrome because it's the browser you'll control from the script.

Install undetected_chromedriver to use it with Selenium and Python.

pip install undetected-chromedriver selenium

Remark: If you don't have Selenium installed, it'll be automatically added with ChromeDriver.

Once we have the tools we'll need, let's write our first lines of code.

import undetected_chromedriver as uc 
 
driver = uc.Chrome() 
driver.get("https://www.nowsecure.nl") 
 
print(driver.current_url) # https://www.nowsecure.nl/ 
print(driver.title) # nowSecure

The script will load the Chrome browser, then redirect and load all the website's resources of our target URL https://www.nowsecure.nl/. To see if it works, we'll print the address and title of this homepage.

It's important to note that base Selenium's WebDriver doesn't come up with the headless mode enabled by default. To change this, we'll pass options to undetected_chromedriver.

# ... 
options = uc.ChromeOptions() 
options.add_argument("--headless") 
driver = uc.Chrome(options=options) 
# ...

Using Undetected ChromeDriver with Selenium

Having established a secure connection with Undetected ChromeDriver, we'll use Selenium to look for the information we want.

We're interested in the information available on the Angular page. So, we'll use CSS selectors and Selenium's find_elements method to match all the occurrences of the class.

from selenium.webdriver.common.by import By 
 
# ... 
driver.get("https://angular.io/") 

node = driver.find_element(By.CSS_SELECTOR, "h1[class='hero-headline no-title no-toc no-anchor']") 
print(node.accessible_name) # Angular - Deliver web apps with confidence

You can check out more use cases of Selenium in our web scraping guide.

Undetected ChromeDriver Proxy

You can also use a proxy with Undetected ChromeDriver to avoid getting blocked while web scraping. However, opting for a free solution won't do you much good because free proxies are often unreliable. The reason is they're run by providers with limited resources and usually outdated infrastructure. And as they're public, many people use them, which can easily result in IP bans.

On the other hand, premium proxies are well-maintained and sourced from reputable ISP providers, so they're a better option for web scraping than datacenter IPs.

Scraping tools like ZenRows come with a pool of premium proxies via API calls, allowing you to easily rotate proxies to hide your IP and avoid all obstacles.

Undetected ChromeDriver User Agent

You can increase your ability to fly under the radar by randomizing the Undetected ChromeDriver User Agent. It's an HTTP header sent with every request to identify the application type, operating system, software version, and other relevant information about the web client (websites often use this data to determine how to respond to requests).

Here's an example of a real User Agent (UA) string:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

However, Selenium's default UA string is easily recognized as a bot for its unique features. Fortunately, Undetected ChromeDriver supports User Agent manipulation.

To change your User Agent in Selenium alongside Undetected Chromedriver, set a new UA string with the variable my_user_agent, and modify the browser options to use the custom UA with options.add_argument.

import undetected_chromedriver as uc
 
# Define a custom user agent
my_user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
 
# Set up Chrome options
options = uc.ChromeOptions()
options.add_argument("--headless")
options.add_argument(f"user-agent={my_user_agent}")
 
# Initialize Chrome WebDriver with the specified options
driver = uc.Chrome(options=options)
 
# Make a request to your target website.
driver.get("https://www.nowsecure.nl/")

# Close the driver
driver.quit()

Additionally, you should dynamically randomize UAs per request for the best results. Check out our guide on changing the Selenium User Agent to learn more.

However, note that more than changing the UA might be required most cases to avoid getting blocked.

Common Errors from Undetected ChromeDriver and Selenium

The most common errors we can get from undetected_chromedriver and Selenium are:

• Denied Access.
• Headless Evasion.

We'll see each in detail to understand how to solve them.

Denied Access

According to the current release on PyPi, Undetected ChromeDriver is more optimized for bypassing anti-bots than Selenium WebDriver. However, the results aren't guaranteed.

For instance, let's try using https://www.zoominfo.com as a target URL.

# ... 
driver.get("https://www.zoominfo.com") 
# Attention Required! | Cloudflare

Here, we fail to bypass the bot detection system implemented by ZoomInfo.

A web scraping API such as ZenRows comes in very handy here because its anti-bot feature successfully handles the bypass for us. As a demonstration, let's use it to scrape our target page.

First of all, get your free API key by signing up, then install ZenRows with this command:

pip install zenrows

As rules, we set antibot and premium_proxy to true before making the API call to set up a connection with the target website.

from zenrows import ZenRowsClient 
 
client = ZenRowsClient(API_KEY) 
url = "https://zoominfo.com" 
params = {"antibot": "true", "premium_proxy": "true"} 
 
response = client.get(url, params=params) 
print(response.text)

ZenRows' antibot feature allows you to bypass the toughest anti-bot solutions out there. That makes it a better-optimized tool compared to Undetected ChromeDriver.

We succeeded in scraping our target URL!

<!DOCTYPE html><html> 
<head> 
	<meta charset="utf-8"> 
	<meta name="viewport" content="width=device-width"> 
	<title>ZoomInfo: B2B Database | Company Contacts & Intelligence</title> 
...

Headless Evasion

According to release 3.1.0 on GitHub, the headless mode is still a work in progress as per a prior release, and no news has been communicated in the latest ones.

That means you shouldn't use headless to have more guarantees to remain undetected. With that in mind, using the driver featured in this article in regular mode might be expensive, depending on your project size.

Module Not Found

The Module Not Found error is a Python exception that logs when the interpreter can't locate the undetected ChromeDriver module you're trying to import. That could be related to incorrect import statements, faulty installation, changes to the package's internal structure, or compatibility issues.

Here are two variants of this error:

modulenotfounderror: no module named 'undetected_chromedriver': This can occur when you incorrectly import the undetected_chromedriver module or the correct version isn't installed for the Python version you have on your machine. To double-check those:

• Ensure you're importing using import undetected_chromedriver as uc.
• Check your Python version using python -V in the terminal and the one of Selenium with pip show selenium. Then, make sure the package is installed for the right versions. For example, Undetected ChromeDriver 3.5.0 only supports Selenium 4.9 and above.

modulenotfounderror: no module named 'undetected_chromedriver.v2': According to release 3.1.0, v2 has become the main module, so there's no need to reference it in your import statement. To fix the error, use import undetected_chromedriver as uc instead of import undetected_chromedriver.v2 as uc.

Best Alternative to Selenium Undetected ChromeDriver

If you still get blocked, there's an alternative. While Selenium with Undetected ChromeDriver can help you minimize detection, you might still be blocked on many occasions, particularly against advanced anti-bot solutions like Cloudflare.

Here's an example of Undetected ChromeDriver against a Cloudflare-protected website, G2 product review page:

import undetected_chromedriver as uc
 
# Create an instance of the undetected ChromeDriver in headless mode
options = uc.ChromeOptions()
options.add_argument("--headless")

driver = uc.Chrome(options=options)
 
# Navigate to target website
driver.get("https://www.g2.com/products/asana/reviews")
 
# Take a screenshot
driver.save_screenshot("screenshot.png")
 
# Close the browser
driver.quit()

We got the following result:

That shows that we were denied access because, ironically, Undetected ChromeDriver was detected by Cloudflare. While you can add proxies or change User Agents to increase your chances, you might still be identified as a bot. Fortunately, web scraping APIs like ZenRows have emerged as the ultimate solution.

ZenRows is an all-in-one web scraping solution that handles all anti-bot bypasses for you, allowing you to focus on getting the data you want.

Let's see it against the previous Cloudflare-protected web page where Undetected ChromeDriver failed. Sign up for free, and you'll get to the Request Builder page:

Input the target URL (https://www.g2.com/products/asana/reviews), activate the "Anti-bot" boost mode, and use "Premium Proxies". Also, select Python as a language.

That'll generate your request code on the right. Copy it and install any HTTP client, like Python Requests, which you can install with the following command:

pip install requests

Your new code should look like this:

import requests
 
url = 'https://httpbin.io/anything'
apikey = 'Your_API_KEY'
params = {
    'url': url,
    'apikey': apikey,
    'js_render': 'true',
    'antibot': 'true',
    'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

Run it, and this will be your result:

<!DOCTYPE html>
 
#...
 
<title>Asana Reviews 2023: Details, Pricing, &amp; Features | G2</title>
 
#...

Bingo! ZenRows makes bypassing any anti-bot solution easy.

Conclusion

In this Undetected ChromeDriver tutorial with Selenium in Python, we learned why and how the library could help us. Also, we compared it with a popular alternative that might fit your needs better in some use cases. 

Here's a quick overview:

Unlike Selenium WebDriver, undetected_chromedriver is more optimized, which makes it better at bypassing bot detection systems. However, it sometimes fails to get around anti-bots, making ZenRows a great alternative.

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn. It looks like this tutorial has helped you. Be someone's hero today: Share on Twitter Share on LinkedIn