Selenium Slow? Discover Why and How to Speed Up

Is your Selenium slow, too? Performance is one of the main challenges when performing browser automation, both in testing and scraping, but there are easy solutions!

Read this guide to find out the best tips and solutions to learn how to make Selenium faster!

Why Is Selenium So Slow Loading Pages?

You might think that the main cause behind why Selenium is so slow is a busy target server. Sure, that's one of the key elements, but not the only one!

The target server plays a role in one of the steps. Yet, you have control over the resources of the machine that runs Selenium. Also, you choose how many requests to make and what operations to perform on each page.

In other words, you can speed up Selenium! Most of the issues have an easy solution. Let's now take a look at those to fix the performance of your scripts.

How to Make Selenium Faster: Best Solutions to Avoid a Slow Selenium

Check out our list of tips, solutions, and workarounds to go from a Selenium that's slow loading pages to a faster one. They'll also help you reduce the resources required, saving you money in cloud servers. Find out more about how much Selenium really costs.

Tip 1: Block Selenium Resources You Don't Need

The average webpage includes many resources, such as images, CSS stylings, fonts, JavaScript scripts, and other assets. According to the Web Almanac report, the average desktop page loads:

• 1,026 KB of images.
• 509 KB of JavaScript.
• 72 KB of CSS.
• 31 KB of HTML.

The time and network bandwidth needed to render a page mainly depends on those resources. The more they weigh, the more the browser takes to retrieve and render them. Thankfully, Selenium lets you set which resources to load and which to ignore.

Images have a significant impact on rendering time as they are the resources that weigh the most. Unless you want to take some screenshots in your scripts, you don't really need them.

Configure the Chrome WebDriver instance to avoid rendering images with the following option:

    options = webdriver.ChromeOptions()
    options.add_experimental_option(
        "prefs", {
            # block image loading
            "profile.managed_default_content_settings.images": 2,
        }
    )
    driver = webdriver.Chrome(
        service=service,
        options=options
    )

You can check out our guide on how to block resources in Selenium.

Here's what a sample page will now look like in Selenium:

This simple trick allows you to cut corners and save rendering time and resources.

Tip 2: Choose Selectors with Better Performance

Web scraping is about selecting HTML elements from a page and extracting data from them. And it turns out that using appropriate locators together with the right methods has a significant impact on performance.

How to speed up Selenium? As for methods, use:

• find_element(): When the locator strategy identifies one and only one element on the page, or you want the first node of a list of similar elements.
• find_elements(): When you need to retrieve a set of DOM nodes with the same selector strategy.

These methods go through the DOM tree to find the desired web elements based on the locators specified. In detail, Selenium offers several locator strategies. The most popular are:

• By.ID: Find elements by the id HTML attribute.
• By.CSS_SELECTOR: Find elements through a CSS selector.
• By.XPATH: Find elements via an XPath expression.
• By.NAME: Find elements by the name HTML attribute.

Each strategy leads to different performance so you need to choose them carefully. A benchmark showed that the most efficient ones are: By.CSS > By.NAME > By.XPATH > By.ID.

Keep in mind that these results depend on the browser in use, your selector goal, and the page structure. On pages with few elements, By.ID might be faster than By.CSS_SELECTOR when looking for a specific element.

Use a Faster Headless Browser to Fight Against a Slow Selenium Selenium can control various browsers, including Chrome, Safari, Firefox, Opera, and Edge. In general, it supports most WebKit-based and Chromium-based browsers. As different browser has different performance, you need to make the right decision.

A recent performance benchmark declares an undisputed winner: Google Chrome.

Not only is Chrome the fastest in all benchmarks, but it is also the most efficient one. No wonder, it has more than 60% of the market share.

Thus, unless you need a specific browser for some requirement, go for the Chrome WebDriver:

    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    
    service = Service()
    options = webdriver.ChromeOptions()
    # your custom options...
    
    driver = webdriver.Chrome(
        service=service,
        options=options
    )

Tip 3: Run Requests in Parallel to Speed up Selenium

When scraping a large site, one can't simply scan each page at a time. Consider that loading a page in a browser can take up to 10 seconds, and you need to multiply that by the number of pages to scrape. In other words, the sequential approach would take forever.

That's why you should configure Selenium to run requests in parallel.

Parallel execution involves launching several Selenium instances to control many browsers simultaneously, which makes the entire scraping process much faster.

Create more than one driver in the same script and handle them in threads to achieve parallelism. Otherwise, adopt Selenium Grid to distribute scripts across multiple machines and browsers.

Tip 4: Reboot Your Servers Programmatically

Rebooting the server hosting your scraping script can be a solution for Selenium's slow performance because the browser instances controlled by the headless browser tend to accumulate temporary files. Also, long-running processes can lead to memory leaks when not releasing resources correctly.

Thus, a reboot provides a fresh environment for running your scraping process. That helps mitigate memory leaks, cuts down disk use, and reduces the risk of slowdowns.

Tip 5: Smart Wait for Elements to Load

The element or data of interest on a page may appear only after a specific interaction. After simulating that interaction, you may stop the execution for a fixed number of seconds to give the page the time to render the desired content:

    # simulate interaction...
    time.sleep(5) # stopping the execution for 5 seconds 
    # select the newly rendered element....

This is a hard wait scenario, and you should always avoid it. The reason? You can't really know the right number of seconds to pause the script for. In most cases, your script will stay idle longer than it should. That's terrible for performance.

Instead, you should use smart waits like WebDriverWait. This allows you to wait for specific conditions to be met before proceeding, making Selenium scripts faster.

For example, the below snippet waits up to 10 seconds for the #card-5 element to become visible on the page. As soon as it's visible, it'll move to the next instruction to avoid idle time.

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    element = WebDriverWait(driver, 10).until(EC.element_to_be_visible((By.ID, "card-51")))

Selenium's alternatives, like Playwright and Puppeteer, have many built-in smart wait methods as well. Playwright even executes implicit wait logic by default.

Tip 6: Make the Most of Your Browser Instance

Most of the resources used by Selenium come from the creation of the browser instance. That's why you should try to make the most of it. Therefore, recycle the browser window whenever you can and avoid closing it earlier than necessary as a best practice.

Faster Alternative to Selenium

If the previous tips to avoid Selenium slow loading pages weren't enough for you, here's an alternative:

ZenRows offers the same JavaScript rendering capabilities as Selenium. The main difference between the two is that ZenRows is faster, more effective, and more powerful than Selenium thanks to its extensive set of features, which include a built-in toolkit to avoid getting blocked while web scraping.

As the ultimate tool for data extraction, it comes with unlimited concurrency and it's also easier to integrate, set up, maintain, and scale.

Getting started with ZenRows is immediate and takes only a few steps:

First of all, sign up to get your free 1,000 credits. You'll reach the Request Builder page.

Now, do as follows:

1. Paste your target URL, activate the “AI Anti-bot” mode (which includes JS rendering), and add “Premium Proxies” for maximum effectiveness.
2. Choose your programming language or cURL to get the API endpoint you can call with any HTTP client. And select the “API” connection mode.
3. Copy the generated code and use it in your script script.

In Python, you'll get the following script:

    # pip install requests
    import requests
    
    url = 'https://httpbin.io/anything'
    apikey = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
    params = {
        'url': url,
        'apikey': apikey,
        'js_render': 'true',
        'antibot': 'true',
        'premium_proxy': 'true',
        # 'js_instructions': your JS instructions...,
    }
    response = requests.get('https://api.zenrows.com/v1/', params=params)
    print(response.text)

Install the request library, and you're ready to run it. ZenRows will take care of running your request in a browser via a proxy while overcoming anti-bot measures. The API response will contain the HTML content of the target page.

Bye-bye Selenium slow performance!

Conclusion

Is Selenium slow? This in-depth guide taught you why and how to fix that. You explored the main workarounds to speed up Selenium, and can now perform browser automation scripts faster than ever!

You know now:

• The reasons behind why Selenium is too slow.
• The main tips to make Selenium faster.
• More efficient alternatives.

Optimizing Selenium is a never-ending process that can make your scripts unstable. Avoid that headache with ZenRows, an all-in-one web scraping API with headless browser capabilities, IP rotation, anti-CAPTCHA, and a whole built-in bypass toolkit to avoid any blocks. Scraping dynamic-content sites effectively and interact with web pages, even at scale, has never been faster. Try ZenRows for free!

driver.set_page_load_timeout(10) # wait up to 10 second for the page to load

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn. It looks like this tutorial has helped you. Be someone's hero today: Share on Twitter Share on LinkedIn