Undetected ChromeDriver in Python Selenium: How to Use for Web Scraping

Are you looking to enhance your Selenium scraper's anti-bot bypass capability? The Undetected ChromeDriver, a Selenium ChromeDriver patch that can evade anti-bot detection, can come in handy.

In this tutorial, you'll learn how Undetected ChromeDriver works, solutions to its common issues and errors,  and how to use it with Selenium in Python.

Let's go!

What Is Undetected ChromeDriver?

Undetected ChromeDriver is an optimized version of the Selenium ChromeDriver designed to prevent anti-bot detection during scraping.

The standard Selenium ChromeDriver presents botlike parameters like the WebDriver properties, which increases the chances of getting blocked. Undetected ChromeDriver patches these loopholes with human-like fingerprints, making your scraper less detectable.

Undetected ChromeDriver boasts of bypassing popular anti-bots, such as Cloudflare, Imperva, and DataDome. It works with Google Chrome, Brave, and many other Chromium browsers. 

How to Use Undetected ChromeDriver in Python With Selenium?

Let's go through a step-by-step tutorial on using Undetected ChromDriver with Selenium, starting with the prerequisites.

First Steps

Before starting with Undetected ChromeDriver, ensure you first install the following:

• Python 3 since the driver works only with Python 3.6 or higher.
• Chrome because it's the browser you'll control from the script.

Once those are ready, install undetected_chromedriver to use it with Selenium and Python:

pip3 install undetected-chromedriver

This installation also includes the standard Selenium library, so you don't need to install Selenium separately:

Once we have the required tools, let's write our first lines of code. 

Open a main check for your scraping logic. Set the use_subprocess parameter to False to run the ChromeDriver as part of the main Python process, reducing memory overhead and increasing the chances of bypassing anti-bot detection. Load the Chrome browser and open the target website: Hapag Lloyd, a website with mild Cloudflare protection. Then, get the website's URL and page title.

import undetected_chromedriver as uc

if __name__ == "__main__":

    # instantiate a Chrome browser
    driver = uc.Chrome(
        use_subprocess=False,
    )

    # visit the target URL
    driver.get("https://www.hapag-lloyd.com/en/home.html")

    # print the URL
    print(driver.current_url)  # https://www.hapag-lloyd.com/en/home.html

    # get the website's title
    print(driver.title)  # Hapag-Lloyd - Global container liner shipping - Hapag-Lloyd

    # close the browser
    driver.quit()

Congratulations! You just bypassed basic Cloudflare security with the Undetected ChromeDriver. 

It's important to note that base Selenium's WebDriver doesn't come up with the headless mode enabled by default. To change this, we'll pass the Chrome options to undetected_chromedriver:

# ... 

if __name__ == "__main__":
   
    # instantiate Chrome options
    options = uc.ChromeOptions()

    # add headless mode
    options.headless = True

    # instantiate a Chrome browser and add the options parameter
    driver = uc.Chrome(
        # ...,
        options=options,
    )
    # ...your scraping logic

That's a basic setup. Let's see how to use Undetected ChromeDriver side-by-side with Selenium to scrape specific elements.

Scrape Data Using Undetected ChromeDriver and Selenium

We're interested in the information on the previous target website. We'll use CSS selectors and Selenium's find_element method to locate the target web elements.

First, import the Byclass from Selenium, visit the target URL, and locate the product's parent element using the find_elements method. Run the browser in non-headless mode to increase the chance of success:

import undetected_chromedriver as uc
from selenium.webdriver.common.by import By

if __name__ == "__main__":

    # instantiate Chrome options
    options = uc.ChromeOptions()

    # add headless mode
    options.headless = False

    # instantiate a Chrome browser and add the options
    driver = uc.Chrome(
        use_subprocess=False,
        options=options,
    )

    # visit the target URL
    driver.get("https://www.hapag-lloyd.com/en/home.html")

    # find the target element
    heading = driver.find_element(By.CSS_SELECTOR, ".hal-h1")

    # print the element text
    print(heading.text) 

    # close the browser
    driver.quit()

Here's the output:

We make Global Trade happen

You can check out more use cases in our Selenium web scraping guide.

Undetected ChromeDriver Proxy

Proxies route your requests through a different IP address, so it looks like your request comes from another machine. You can use a proxy with Undetected ChromeDriver to avoid anti-bot measures like IP bans and geo-restrictions while web scraping.

There are two proxy categories based on cost: free and premium. However, free proxies are unsuitable for real-life projects due to their short lifespan. Additionally, they get detected easily, as many people use them simultaneously, resulting in IP bans.

On the other hand, premium proxies are well-maintained and sourced from reputable internet service providers (ISPs), so they're a better option for web scraping. They guarantee quality, secure, and reliable connections. Some providers even offer advanced features like auto-rotation, geo-location, and dedicated anti-bot bypass tools.

ZenRows is one of the top premium proxy providers, offering 55+ million residential IPs globally distributed across 185+ countries. It provides advanced scraping utilities, including CAPTCHA and anti-bot auto-bypass, proxy auto-rotation, geo-location, and more out of the box.

Read our detailed tutorial on setting Undetected ChromeDriver proxies to learn more about adding proxies to Undetected ChromeDriver.

Undetected ChromeDriver User Agent

You can increase your ability to fly under the radar by randomizing the Undetected ChromeDriver User Agent. It's an HTTP header sent with every request to identify the application type, operating system, software version, and other relevant information about the web client (web servers often use this data to determine how to respond to requests).

Undetected ChromeDriver's default User Agent contains a bot-like signal like the HeadlessChrome, making it prone to anti-bot detection. See what it looks like below:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/127.0.0.0 Safari/537.36

Fortunately, Undetected ChromeDriver allows you to replace the default User Agent with a custom one. It's best to randomize several User Agents from a list to make your requests more legitimate.

Check out how to add User Agents to Undetected ChromeDriver to learn more.

Common Errors from Undetected ChromeDriver and Selenium

There are a few common errors that may occur while using Undetected ChromeDriver. Let's quickly see three of them.

Denied Access

According to the current release on PyPi, Undetected ChromeDriver is more optimized for bypassing anti-bots than Selenium WebDriver. However, the results aren't guaranteed.

For instance, let's try using a protected website like OpenSea as a target website:

if __name__ == "__main__":

    # ...

    # visit the target URL
    driver.get("https://opensea.io/")
    # print the page source
    print(driver.page_source) # Access denied

The scraper returns an "Access denied" error, showing that it got blocked by an anti-bot. 

The best solution to bypass that block is to use a web scraping API such as ZenRows. It's an all-in-one solution that helps you auto-rotate premium proxies, fix your request headers, bypass CAPTCHAs and any other anti-bot at scale. ZenRows also has headless browsing features, making it an excellent alternative to Undetected ChromeDriver.

Headless Evasion

According to release 3.1.0 on GitHub, the headless mode is still a work in progress as per a prior release, and news has yet to be communicated in the latest ones. For example, when you use the library in headless mode, its default User Agent header has a HeadlessChrome flag that makes it obvious that you're using an automated program.

That means you should use Undetected ChromeDriver in non-headless (GUI) mode to have more guarantees of remaining undetected. Utilizing the driver featured in this article in regular mode might be expensive, depending on your project size.

Module Not Found

The Module Not Found error is a Python exception that logs when the interpreter can't locate the undetected ChromeDriver module you're trying to import. That could be related to incorrect import statements, faulty installation, changes to the package's internal structure, or compatibility issues.

Here are two variants of this error:

1. modulenotfounderror: no module named 'undetected_chromedriver': This can occur when you incorrectly import the undetected_chromedriver module or the correct version isn't installed for the Python version you have on your machine. To double-check those:

• Check your Python version using python -V in the terminal and ensure it's Python 3.6+.
• Re-install the library using pip3 install undetected-chromedriver.

2. modulenotfounderror: no module named 'undetected_chromedriver.v2': According to release 3.1.0, v2 has become the main module, so there's no need to reference it in your import statement. To fix the error, import it as undetected_chromedriver as uc instead of using undetected_chromedriver.v2 as uc.

Undetected ChromeDriver Disadvantages and Issues

Despite the functionalities of Undetected ChromDriver, it still has some shortcomings you should know.

Undetected ChromeDriver is Easy to Get Detected and Blocked

Although Undetected ChromeDriver is designed to bypass blocks, it still can't deal with sophisticated detection techniques such as advanced browser fingerprinting, machine learning, and rate-limited IP bans.

As a headless browser, Undetected ChromeDriver exhibits certain bot-like characteristics. For instance, the automated WebDriver navigator property might still be detectable despite attempts to mask it. As mentioned, it also shows traces of headless browsing like HeadlessChrome. These properties make it easily detectable by anti-bots. 

Stability Issues of Undetected ChromeDriver

Another limitation of Undetected ChromeDriver is its struggle to keep up with websites' frequently evolving anti-bot systems. As these detection mechanisms become more sophisticated, the bypassing techniques used by Undetected ChromeDriver can become unreliable. 

Additionally, updates to Undetected ChromeDriver are often tied to Chrome version releases, and its automation capabilities depend heavily on the underlying Selenium framework. This dependency can lead to delays in adapting to new challenges and maintaining the effectiveness of its anti-detection features.

Undetected ChromeDriver Is Costly at Scale

Undetected ChromeDriver's dependence on Selenium means you're still subject to potential memory overhead associated with running browser instances. Besides, the library is most effective in GUI mode (non-headless), which necessitates managing multiple browser instances, a task that becomes challenging and unsustainable at scale.

While you can add more computing power to handle the increased memory demand, this approach can become costly when scaling up.

Best Alternative to Selenium Undetected ChromeDriver

If Undetected ChromeDriver doesn't unblock your scraper, there are more reliable alternatives you can try.

Selenium with Undetected ChromeDriver can help minimize chances of detection, but you'll likely still get blocked when bypassing advanced anti-bot solutions like Cloudflare.

Here's an example of Undetected ChromeDriver against a Cloudflare-protected website, G2 product review page:

import undetected_chromedriver as uc

if __name__ == "__main__":

    # instantiate Chrome options
    options = uc.ChromeOptions()

    # add headless mode
    options.headless = False

    # instantiate a Chrome browser and add the options
    driver = uc.Chrome(
        use_subprocess=False,
        options=options,
    )

    # visit the target URL
    driver.get("https://www.g2.com/products/asana/reviews")

    # take a screenshot
    driver.save_screenshot("screenshot.png")

    # close the browser
    driver.quit()

We got the following result:

That shows that Cloudflare ironically detected the Undetected ChromeDriver. If Undetected ChromeDriver doesn't work, try the following alternatives:

• Nodriver: A derivative of Undetected ChromeDriver that doesn't depend on Selenium or its ChromeDriver binary but uses your regular web browser.
• SeleniumBase: A testing and crawling library that applies the best practices for using Undetected ChromeDriver as a bypass plugin.

Check out the best alternatives to Undetected ChromeDriver to learn more.

While these alternatives can increase your chances of bypassing anti-bots, they can still fall short and never guarantee a 100% success rate. The only surefire solution is using a web scraping API, such as ZenRows.

ZenRows is easy to use, compatible with all programming languages, and bypasses any anti-bot system at scale. It's a complete anti-bot detection toolkit that will let you automate your web scraping process and extract data uninterrupted.

Conclusion

In this Undetected ChromeDriver tutorial with Selenium in Python, you learned why and how the library could help. Unlike Selenium WebDriver, Undetected Chromedriver is more optimized, which makes it better at bypassing bot detection systems.

However, it sometimes fails to get around anti-bots, making ZenRows a much bette ralternative. Here's a quick summary of why ZenRows might fit your needs better in most use cases:

Try ZenRows for free now without a credit card!