Using --disable-blink-features=AutomationControlled to Reduce Detection in Headless Chrome

Automated headless browsers are powerful scraping tools, but are prone to anti-bot detection measures. The --disable-blink-features=AutomationControlled flag, a feature associated with Chromium browsers, can help you reduce detection.

This article shows you how to use that flag to hide the WebDriver automation property in Selenium and Playwright.

Let's start with what the --disable-blink-features=AutomationControlled is.

What Is --disable-blink-features=AutomationControlled?

--disable-blink-features=AutomationControlled is a Blink engine flag that prevents Chromium browsers from automatically setting navigator.webdriver to true.

Many anti-bot systems check for WebDriver browser automation as part of their browser fingerprinting techniques. This feature prevents anti-bots from identifying bot-like behavior via the WebDriver.

While automated headless browsers mimic a real browser environment, navigator.webdriver=true is one of the properties that distinguishes them from regular browsers, where it is false by default.

To reduce the risk of detection, headless browser tools like Selenium and Playwright should modify this property. You'll see how to go about it in the next section.

Use the --disable-blink-features=AutomationControlled Flag in Headless Browsers

You can deactivate Blink's automation flag for different headless browsers. You'll learn how to achieve this in Selenium and Playwright. You'll test the changes on SannySoft, a website that analyzes your browser for basic fingerprints.

Each headless browser might return a different result for the other parameters. However, both headless browsers will fail the WebDriver test.

Modifying the WebDriver's visibility with the --disable-blink-features=AutomationControlled ensures the test passes. Let's see how to achieve that in each headless browser.

Selenium

To disable Blink's automation flag in Selenium, include the --disable-blink-features=AutomationControlled flag as a ChromeDriver option while setting up Selenium.

Here's the code to achieve that:

# pip3 install selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# set up Chrome options without Blink's automation flag
chrome_options = Options()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(options=chrome_options)

# open the target page
url = "https://bot.sannysoft.com/"
driver.get(url)

# take a screenshot
driver.save_screenshot("screenshot.png")

# close the browser
driver.quit()

The above code returns a screenshot with a passed WebDriver check:

Let's see how it works in Playwright.

Playwright

You can disable the WebDriver automation flag in Playwright by adding --disable-blink-features=AutomationControlled to the Chromium instance argument:

# pip3 install playwright
# playwright install
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    # set up Chrome options without Blink's automation flag
    browser = p.chrome.launch(args=["--disable-blink-features=AutomationControlled"])

    # open the target page
    page = browser.new_page()
    page.goto("https://bot.sannysoft.com/")

    # take a screenshot
    page.screenshot(path="screenshot.png")

    # close the browser
    browser.close()

Among other failed parameters, Playwright passed the WebDriver test:

Next, you'll see how to modify the webdriver.navigator in Puppeteer.

Puppeteer

Puppeteer also accepts the --disable-blink-features=AutomationControlled flag as an option during the browser instance setup. Include it as shown to disable the WebDriver automation property:

// npm install puppeteer
const puppeteer = require('puppeteer');

(async () => {
    // set up Puppeteer without Blink's automation flag
    const browser = await puppeteer.launch({
        headless: true,
        args: ['--disable-blink-features=AutomationControlled'],
    });

    // open the target page
    const page = await browser.newPage();
    await page.goto('https://bot.sannysoft.com/');

    // take a screenshot
    await page.screenshot({ path: 'screenshot.png' });

    await browser.close();
})();

The above code shows the WebDriver is missing, as shown:

You've just modified the value of webdriver.navigator from true to false, reducing the chances of anti-bot detection.

However, this method alone does not effectively circumvent anti-bot measures, as it only addresses one of many potential bot signals. For instance, Playwright failed in other aspects despite modifying it to pass the WebDriver check.

How can you patch all these loopholes and scrape without getting blocked?

How to Permanently Avoid Getting Blocked While Scraping

Deactivating the Blink automation flag alone doesn't stop you from getting blocked. To access web pages confidently without blocks, you need to patch all possible bot signals.

You can manually modify these headless browsers' properties to evade blocks, but this is often technical, time-consuming, and unreliable. Selenium, Playwright, and Puppeteer also have third-party stealth plugins. However, these are open-source and usually fail to keep up with the ever-evolving anti-bot security measures.

The easiest and most reliable way to avoid getting blocked is to use a web scraping API like the ZenRows Universal Scraper API. With a single API call, ZenRows applies advanced evasion mechanisms required to scrape any website successfully without limitations.

It features request header optimization, premium proxy rotation with geo-location, JavaScript rendering support, AI-powered anti-bot auto-bypass, and many more.

ZenRows also has headless browser features, making it an excellent replacement for open-source options.

Let's quickly see how the ZenRows Universal Scraper API works by scraping a heavily protected site like this Antibot Challenge page.

Sign up on ZenRows to open the Request Builder. Then, paste your target URL in the address box and activate Premium Proxies and JS Rendering.

Select your preferred programming language (Python, in this case) and choose the API connection mode. Copy the generated code and paste it into your scraper.

The generated Python code should look like this:

# pip3 install requests
import requests

url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)

The above code bypasses the anti-bot challenge and outputs the protected site's full-page HTML:

<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉 Your scraper now has the stealth required to extract content from any website without limitations. You'll never get blocked while web scraping again!

Conclusion

You've learned how the --disable-blink-features=AutomationControlled flag works and how to use it to remove the WebDriver automation flag in Selenium, Playwright, and Puppeteer.

Remember that you can't rely on that approach alone, and avoiding anti-bot detection requires full-scale stealth. We recommend using ZenRows, an all-in-one web scraping toolkit, to evade all blocks during web scraping.

Try ZenRows for free!