Browser Fingerprinting: How to Bypass it During Web Scraping

Websites use browser fingerprinting to gather information about your device to profile your browser. But it's a security measure that might block you while web scraping.

In this article, you'll learn what browser fingerprinting is, how it works, and how to avoid it during web scraping.

What Is Browser Fingerprinting?

Browser fingerprinting is a technique for uniquely identifying web clients (browsers and, in some cases, HTTP clients). It involves analyzing parameters such as device attributes, HTTP connection details (e.g., IP address), and software configurations (e.g., browser settings, installed plugins, system fonts).

The collected data points are generic but unique to each web client when combined. So, it's unlikely that two users will have 100% identical data points.

Even a slight variation in attributes (such as screen resolution, installed extensions, or hardware configuration) can lead to profile differences. For example, millions of users visit Google using Chrome. But it's unusual to find multiple people using Chrome with the same extension on a 1366x768 resolution HP OMEN PC with 16 GB RAM and a 4GB Nvidia graphics card on driver version 25.20.99.9221.

Browser fingerprinting uses this combination of unique attributes (and configurations) to create a profile that allows servers or services to identify and track individual web clients.

That said, different browsers/clients have different levels at which they can be fingerprinted. Some are highly vulnerable, while others maintain moderate to high resistance to fingerprinting.

Browser Fingerprinting vs. Cookie Tracking

Browser fingerprinting is different from cookie tracking. Cookie tracking is a conventional method for identifying and monitoring web users by storing data on the client's device. Browser fingerprinting, on the other hand, collects information about the user's browser and device configuration without requiring local storage.

Cookies are small data packets sent to a user's browser by a web server. The browser stores the packet and reuses it to identify the user during subsequent visits. Users have some control over this tracking method, as websites often request consent to store cookies. Deleting or blocking cookies prevents websites from tracking your activity.

In contrast, users have no such control over browser fingerprinting. When a browser makes a request, data is automatically collected to create a unique profile, allowing websites to identify users without their consent.

Let's see how websites access the data needed to generate a fingerprint in more detail.

How Does Browser Fingerprinting Work?

One primary application of browser fingerprinting is to profile clients and assess their authenticity. Unfortunately, this technique often flags web scrapers as bots because they don't resemble legitimate browsers.

When you visit a page, your web client sends a request to the site's server with the data required to establish a connection. As noted earlier, this information may include your IP address, browser properties, and other details.

Some data reaches the web server with the first connection packet. However, in most cases, the website injects JavaScript to generate more granular fingerprint data, depending on the technique used. These scripts run in the background and are difficult to identify due to obfuscation.

Typically, each web client has different values for the data points these scripts query, allowing websites to create unique fingerprints and identify each user. For instance, web scrapers generate bot-like fingerprints, which can result in access denied or a 403 forbidden error during web scraping.

Cross-Browser Fingerprinting

Cross-browser fingerprinting is a method for identifying a user across multiple browsers on the same machine. Cross-browser fingerprinting enables websites to identify you, even if you try to evade detection by switching browser types, versions, or vendors.

The cross-browser fingerprinting technique relies primarily on collecting consistent information across all fingerprinted browsers, such as fonts, color depth, screen resolution, CPU class, device memory, operating system version, and more.

Cross-browser fingerprinting can be more challenging to bypass than single-browser fingerprinting, especially if you rely on spoofing multiple browser fingerprints.

Before we discuss the various fingerprinting techniques, let's analyze real-world examples to understand how they work.

Browser Fingerprinting Example

As a first example, we'll review a JavaScript file from Keywee, a platform that uses natural language processing and machine learning to help publishers and marketers create, distribute, and measure the performance of their content:

The above script is minified, so deducing what it does is difficult. However, we can use an online prettifier to make the code easier to understand. Here's the result.

Starting from line 1415, you'll see functions for collecting data points.

// ...





   17: [function(b, c, a) {


       (function() {
            var l = b("../lib_managed/lodash"),
                k = b("murmurhash").v3,
                g = b("jstimezonedetect").jstz.determine(),
                e = b("browser-cookie-lite"),
                h = typeof a !== "undefined" ? a : this,
                j = window,
                d = navigator,
                i = screen,
                f = document;


           h.hasSessionStorage = function() {


               try {


                   return !!j.sessionStorage


               } catch (m) {


                   return true


               }


           };


           h.hasLocalStorage = function() {


               try {


                   return !!j.localStorage


               } catch (m) {


                   return true


               }


           };


           h.localStorageAccessible = function() {


               var m = "modernizr";


               if (!h.hasLocalStorage()) {


                   return false


               }


               try {


                   j.localStorage.setItem(m, m);


                   j.localStorage.removeItem(m);


                   return true


               } catch (n) {


                   return false


               }


           };


           h.hasCookies = function(m) {


               var n = m || "testcookie";


               if (l.isUndefined(d.cookieEnabled)) {


                   e.cookie(n, "1");


                   return e.cookie(n) === "1" ? "1" : "0"


               }


               return d.cookieEnabled ? "1" : "0"


           };





       // ...

The code snippet retrieves fingerprint components into the variable p on line 1461, as shown below. The function k on line 1475 returns an integer hash code for the fingerprint components.

      // ...

        h.detectSignature = function(r) {
            var p = [d.userAgent, [i.height, i.width, i.colorDepth].join("x"), (new Date()).getTimezoneOffset(), h.hasSessionStorage(), h.hasLocalStorage()];
            var m = [];
            if (d.plugins) {
                for (var q = 0; q < d.plugins.length; q++) {
                    if (d.plugins[q]) {
                        var n = [];
                        for (var o = 0; o < d.plugins[q].length; o++) {
                            n.push([d.plugins[q][o].type, d.plugins[q][o].suffixes])
                        }
                        m.push([d.plugins[q].name + "::" + d.plugins[q].description, n.join("~")])
                    }
                }
            }
            return k(p.join("###") + "###" + m.sort().join(";"), r)
        };

        // ...

In this case, the collected data includes:

• User-Agent string.
• Window size.
• Color.
• Time zone.
• Plugins.

As in the example above, most websites use JavaScript files to identify and isolate web clients.

You'll now see a real-life example by locating the browser fingerprinting script on Le Monde, a French news website.

First, open the Performance tab in the developer tools (right-click, select Inspect, then go to Performance). Navigate to the call tree and locate the first function call:

Open this script in a new tab by clicking the highlighted link in the image above.

That said, let's beautify this script to understand its purpose better. The complete script is available in this GitHub Gist.

On lines 1038 and 1042, you'll see data points stored in variables e and p. They include:

• User-Agent.
• Browser vendor.
• Window size.
• Browser name.

// Window width and height.


// ...





f = () => {


   const e = navigator.userAgent || navigator.vendor || window.opera;
    return -1 !== e.indexOf("FBAN") || -1 !== e.indexOf("FBAV")
},
p = () => window.innerWidth || document.documentElement.clientWidth || document.body.clientWidth;


let m = !1;


const h = () => {


   const e = sessionStorage.getItem("pageinsession");


   if (!0 === m && e) return e.length;


   if (m = !0, "reload" !== d()) {


       const t = e && "".concat(e, "i") || "i";


       return sessionStorage.setItem("pageinsession", t), t.length


   }


   return e ? e.length : 1


},





// ...

Following the same principle, other functions may collect additional fingerprint data in this script.

What Information Is Collected During Browser Fingerprinting?

As shown in the previous example, commonly collected data points in browser fingerprinting include User-Agent strings, window size, browser plugins, vendor, time zone, and system properties. Websites that employ anti-bot measures may collect additional attributes depending on their configuration.

Although there's no universal standard for what data a website collects, typical data points include:

• Operating system type and language.
• Browser type, version, and extensions.
• Screen resolution and color depth.
• Time zone and fonts.
• Language and keyboard layout.
• User-Agent string and navigator properties.

Advanced scraping techniques involve patching these fingerprints to reduce the chances of anti-bot detection. That said, you can use a tool like CreepJS to identify collected data points, allowing you to understand which attributes may need patching to evade browser fingerprinting techniques.

Fingerprinting Techniques

Sites use multiple methods to interact with web clients' properties to access and identify user data. Let's look at some of them!

Canvas Fingerprinting

The HTML5 Canvas is an API that provides built-in objects, methods, and properties for drawing text and graphics on a canvas. Browsers use their features to render web content.

In 2012, Mowery and Shacham found that the final user-visible canvas graphic is directly affected by several factors, including:

• Operating system.
• Browser version.
• Graphics card.
• Installed fonts.
• Sub-pixel hinting.
• Antialiasing.

The interpretation is that the generated images and text vary based on the device's graphics capabilities. As a result, websites that use browser fingerprinting request that web clients render canvas images. They can then collect and store this information in their browser fingerprint database.

Check out our in-depth article on canvas fingerprinting to learn more.

WebGL Fingerprinting

Like HTML5 Canvas, WebGL is a graphics API for rendering 3D interactive images. These graphics are rendered differently based on the device's graphics capabilities. So, websites that use browser fingerprinting can task a browser's WebGL API with rendering 3D images to extract unique device features from the results.

AudioContext Fingerprinting

The Web Audio API is a powerful interface for generating and processing audio. It connects audio nodes to apply effects like compression and filtering. AudioContext fingerprinting operates on the principle that devices process audio signals differently due to variations in hardware and software configurations, including audio drivers and browser implementations.

Similar to Canvas and WebGL fingerprinting, this technique leverages these differences to generate a unique fingerprint for tracking. Since AudioContext fingerprinting is still relatively novel, few websites implement Web Audio API scripts.

Font Fingerprinting

One unpopular but effective browser fingerprinting technique is to identify users based on rendered text. The principle here is that browsers on different devices render the same font style and character with different bounding boxes. Consequently, sites can query these bounding boxes to collect fingerprinting data.

Battery Fingerprinting

Battery fingerprinting involves tracking a device by leveraging information about its battery status or behavior. Using the Battery Status API, websites can access attributes like charge level, charging status (charging or discharging), and the estimated time for a full charge or discharge.

These attributes often vary due to device usage and battery health differences, making them potential components for creating unique identifiers in browser fingerprinting.

Media Device Fingerprinting

Media device fingerprinting collects information about the user's available audio and video hardware via the MediaDevices API. Data collected can include:

• The number and types of connected media devices, including cameras, microphones, and speakers.
• Device IDs, which are typically anonymized.
• Supported media constraints like frame rates, resolutions, and audio characteristics.

Even if device IDs are anonymized, other attributes, such as supported formats or connected devices, can still contribute to building a browser fingerprint.

How to Plug Fingerprint Leaks?

We can plug fingerprint leaks by imposing fake values whenever the property is accessed. For example, the script below plugs the navigator.webdriver leak.

Object.defineProperty(navigator, 'webdriver', {
    get: function () {
        return false;
    },
});

The code above uses Object.defineProperty() to modify a property on the navigator object. It then uses the GET function to modify the underlying behavior when the server queries the browser's navigator.webdriver property. In this case, the function will always return false.

You can add this code to the beginning of your web scraping script. That way, when the website checks the navigator.webdriver value, it'll always return false and won't be able to detect that the browser is automated.

Browser fingerprinting alone is quite powerful for detecting web scrapers, and headless browsers can make fingerprint identification easier. They set default properties that flag them as bots in the JavaScript execution context.

By detecting these property-value pairs, websites can identify and block web scrapers that use headless browsers such as Selenium and Playwright. Therefore, making your scraping environment undetectable should be the first step to bypassing browser fingerprinting.

Let's see how to do that for the most popular headless browsers:

Selenium: You can execute the script using the execute_script() method. Here's an example of how to do it in Python to bypass browser fingerprinting:

# pip3 install selenium
from selenium import webdriver

# create a new instance of the Chrome driver
driver = webdriver.Chrome()

# execute the script to hide the fact that the browser is automated
driver.execute_script(
    "Object.defineProperty(navigator, 'webdriver', {get: function() {return false}})"
)

Playwright: You can use the evaluate() method to execute the script in Python. Check the following example:

# pip3 install playwright
# playwright install
from playwright.sync_api import sync_playwright


with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    page = context.new_page()

    # execute the script to hide the fact that the browser is automated
    page.evaluate("Object.defineProperty(navigator, 'webdriver', {get: () => false});")

    # close the browser after execution
    browser.close()

Puppeteer: Also uses the evaluate() method to execute the script. See the example implementation in JavaScript below:

// npm install puppeteer
const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    // execute the script to hide the fact that the browser is automated
    await page.evaluate(() => {
        Object.defineProperty(navigator, 'webdriver', { get: () => false });
    });
})();

To learn more about plugging Puppeteer leaks, read our comprehensive article on patching Puppeteer Stealth for anti-bot bypass.

Patching leaks in headless browsers, such as Selenium, Playwright, and Puppeteer, makes your requests appear more natural. Unfortunately, this technique is usually insufficient against Web Application Firewalls (WAFs) such as Cloudflare and DataDome, which employ advanced techniques to detect these patches.

So, what's the solution?

How to Bypass Browser Fingerprinting

Bypassing browser fingerprinting can prove challenging because you can't accurately predict the data an anti-bot collects. Even with custom evasions, you can still get blocked.

Since websites use scripts to collect and create fingerprints, disabling JavaScript is one way to avoid browser fingerprinting. However, that approach is unreliable because it affects usability and content availability, since most websites today rely on JavaScript to render content.

Use a Free Stealth Browser

Steath browsers are enhanced versions of standard headless browsers. Instead of manually patching headless browsers such as Selenium, Playwright, and Puppeteer with custom evasions, stealth versions include built-in bypass techniques to fortify the main headless browser to bypass anti-bot detection.

These stealth headless browser versions can cover other JavaScript-detectable features you might omit during custom patching. Fortunately, each popular headless browser automation tool has a stealth version or plugin:

• SeleniumBase with Undetected ChromeDriver for Selenium.
• Playwright Stealth plugin for Playwright.
• Puppeteer Stealth plugin for Puppeteer.

While open-source stealth headless browsers can significantly increase your chances of bypassing anti-bots, they're also unreliable in edge cases. Additionally, because they are open-source, these tools can't keep up with the ongoing updates to anti-bot measures.

Bypass Browser Fingerprinting With a Web Scraping API

The easiest way to bypass browser fingerprinting is to use a web scraping API, such as the ZenRows Universal Scraper API.

ZenRows includes all the capabilities required for successful scraping, including advanced fingerprinting spoofing, premium proxy rotation, request header management, anti-bot auto-bypass, JavaScript rendering, and more.

ZenRows also includes headless browser capabilities, allowing you to interact with a web page as a human user. Its Adaptive Stealth Mode provides all the necessary configuration for success at the lowest possible cost. This allows you to focus on core data cleaning and refinement rather than wasting time and resources patching detection leaks. All these features are available via a single API call, in any programming language.

Let's see how the ZenRows Universal Scraper API works by scraping the full-page HTML of the Antibot Challenge page.

Sign up and open Request Builder. Then paste the target URL into the link box and enable Adaptive Stealth Mode.

Select your programming language (Python, in this case) and choose the API connection mode. Copy and paste the generated code into your Python script.

Here's the generated Python code:

# pip3 install requests
import requests

url = 'https://www.scrapingcourse.com/antibot-challenge'
apikey = '<YOUR_ZENROWS_API_KEY>'

params = {
    'url': url,
    'apikey': apikey,
    'mode': 'auto',
}

response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

The above scraper accesses the protected website and scrapes its full-page HTML, as shown:

<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉. You just found a reliable way to bypass anti-bot detection using ZenRows.

Conclusion

You've seen how browser fingerprinting works and how to bypass it, including custom and open-source techniques. However, browser fingerprinting is a powerful anti-bot detection technique, and those methods are unreliable at scale.

We recommend bypassing all blocks with an all-in-one web scraping solution like ZenRows and spare yourself the time and effort of dealing with regular anti-bot system updates.

Try ZenRows for free now or speak with sales!