Puppeteer Fingerprinting: Explained & Bypassed

Is your Puppeteer scraper getting blocked? Modifying its fingerprints can improve stealth.

In this article, you'll learn how Puppeteer fingerprinting works, how it contributes to anti-bot detection, and how to improve its fingerprints to bypass blocks during scraping. Let's begin.

What Is Puppeteer Fingerprinting?

Puppeteer fingerprinting isn't a standard term but is part of browser fingerprinting techniques for profiling and detecting Puppeteer automation. 

The browser fingerprinting process distinguishes devices or users based on unique properties, such as IP address, browser configurations, device type, graphic renderers, plugins, screen resolution etc. These parameters are usually unique to each user, and websites leverage them to identify the client sending a request at any time.

Why Puppeteer Fingerprinting Is a Threat to Web Scraping

Anti-bots like Cloudflare, PerimeterX, and DataDome may not actively implement Puppeteer fingerprinting. However, their bot detection algorithm snoops for bot-like patterns from browser automation tools like Puppeteer.

One method that anti-bot systems use to detect automated access is to compare a client's fingerprints to a database of known bots. If the fingerprint matches a known bot, the system can block that traffic to prevent further access.

Missing or inconsistent fingerprint parameters can also result in anti-bot detection. As an open-source browser automation tool, Puppeteer often presents distinct bot characteristics, such as the presence of navigator.webdriver=true and HeadlessChrome automation flags. That said, you can avoid detection in Puppeteer by improving it with custom evasion patches, spoofing real fingerprints, using personalized settings, and adding stealth plugins.

In the next section, you'll apply a Puppeteer fingerprint plugin to reduce the chances of getting blocked during web scraping. 

What Is puppeteer-with-fingerprints and How Does It Work?

puppeteer-with-fingerprints is a plugin for modifying the browser fingerprint in Puppeteer to improve stealth and reduce the chances of detection. It obtains browser data from the FingerprintSwitcher server, which lets you spoof different browser fingerprints.

The plugin allows you to change browser properties like plugins, WebGL, platform, browser type and version, screen resolution, audio codec, etc. It also supports proxy mode, allowing you to switch IPs or choose a specific geolocation.

You can plug puppeteer-with-fingerprints into an existing or new scraper. You'll learn how to use this tool in the next section.

Bypass Browser Fingerprinting Using puppeteer-with-fingerprints During Scraping

In this section, you'll use puppeteer-with-fingerprints to modify the browser fingerprints in a Puppeteer scraper. You'll then test the result on Sannysoft, a website that returns basic browser fingerprint properties to determine if your request is bot-like. 

To use the puppeteer-with-fingerprints plugin, you must install it alongside Puppeteer using npm:

npm install puppeteer puppeteer-with-fingerprints

For a benchmark template, let's see what the base Puppeteer library returns when tested for fingerprinting on the target test website. Try it with the following code:

// npm install puppeteer
const puppeteer = require('puppeteer');

(async () => {
    // launch the browser in headless mode
    const browser = await puppeteer.launch({ headless: true });
    const page = await browser.newPage();

    // open the target page
    await page.goto('https://bot.sannysoft.com/', {
        waitUntil: 'networkidle2',
    });

    // take a screenshot
    await page.screenshot({ path: 'screenshot.png' });
})();

The base Puppeteer library fails the WebDriver and HeadlessChrome checks as shown:

An anti-bot measure is likely to block the above scraper. To modify it with browser fingerprints, proceed to the next steps.

Step 1: Configure the Fingerprint Service

The first step is to initialize puppeteer-with-fingerprints in your Puppeteer scraper. This plugin replaces the base Puppeteer library. 

To set it up, import the library into your Puppeteer scraper. Then, implement a plugin key using an empty string. The empty string in the plugin.setServiceKey function is the default key to activate the free version of the underlying tool (FingerPrintSwitcher).

// npm install puppeteer puppeteer-with-fingerprints
const { plugin } = require('puppeteer-with-fingerprints');

// set up the plugin key
plugin.setServiceKey('');

That was easy! You're ready to apply browser fingerprints to your scraper.

Step 2: Fetch and Apply Fingerprints

The plugin inherits Puppeteer's API, giving you the dual benefit of optimizing browser fingerprints while automating user interactions.  

To apply fingerprints from the plugin, start an anonymous asynchronous function and fetch the desired browser fingerprints. Add the fingerprint configuration to the plugin instance and use it to launch a browser in headless mode:

// ...

(async () => {
    // fetch the desired browser fingerprint
    const fingerprint = await plugin.fetch({
        tags: ['Microsoft Windows', 'Chrome'],
    });

    // add the fingerprint to the plugin
    plugin.useFingerprint(fingerprint);

    // launch the browser in headless mode
    const browser = await plugin.launch({ headless: true });
    const page = await browser.newPage();

})();

Your Puppeteer scraper now uses a regular Chrome browser's fingerprint on a Windows operating system. In the next section, you'll apply the fingerprint in a request.

Step 3: Test the Fingerprint Behavior

It's time to test if the applied fingerprint works. To do that, you'll request and screenshot the Sannysoft web page to view the returned fingerprint parameters.

Expand the previous snippet to open the target website and take its screenshot:

// ...
(async () => {
    // ...

    // open the target page
    await page.goto('https://bot.sannysoft.com/', {
        waitUntil: 'networkidle2',
    });

    // take a screenshot
    await page.screenshot({ path: 'screenshot.png' });
})();

Merge the snippets, and you'll get the following complete code:

// npm install puppeteer puppeteer-with-fingerprints
const { plugin } = require('puppeteer-with-fingerprints');

plugin.setServiceKey('');

(async () => {
    // fetch the desired browser fingerprint
    const fingerprint = await plugin.fetch({
        tags: ['Microsoft Windows', 'Chrome'],
    });

    // add the fingerprint to the plugin
    plugin.useFingerprint(fingerprint);

    // launch the browser in headless mode
    const browser = await plugin.launch({ headless: true });
    const page = await browser.newPage();

    // open the target page
    await page.goto('https://bot.sannysoft.com/', {
        waitUntil: 'networkidle2',
    });

    // take a screenshot
    await page.screenshot({ path: 'screenshot.png' });
})();

The puppeteer-with-fingerprint plugin passes all checks, proving you've patched your Puppeteer scraper with an actual browser fingerprint:

Bravo! Your Puppeteer scraper just got stealthier. 

However, despite being a powerful browser fingerprint spoofing tool, the plugin still has some limitations.

Limitations of the puppeteer-with-fingerprints Plugin

Spoofing browser fingerprints can increase the likelihood of evading anti-bots, but the technique doesn't cover all detection measures. It might also add suspicious fingerprints. As an example, the User Agent's Chrome version in the above result is outdated and might get flagged.

For instance, the current Puppeteer scraper won't pass on a heavily protected website like the Antibot Challenge page. Try it out with the following code:

// npm install puppeteer puppeteer-with-fingerprints
const { plugin } = require('puppeteer-with-fingerprints');

plugin.setServiceKey('');

(async () => {
    // fetch the desired browser fingerprint
    const fingerprint = await plugin.fetch({
        tags: ['Microsoft Windows', 'Chrome'],
    });

    // add the fingerprint to the plugin
    plugin.useFingerprint(fingerprint);

    // launch the browser in headless mode
    const browser = await plugin.launch({ headless: true });
    const page = await browser.newPage();

    // open the target page
    await page.goto('https://www.scrapingcourse.com/antibot-challenge', {
        waitUntil: 'networkidle2',
    });

    // take a screenshot
    await page.screenshot({ path: 'screenshot-anb.png' });
})();

The scraper got blocked, as shown:

Websites use other anti-scraping mechanisms, including request header monitoring, behavioral analysis, JavaScript challenges, rate limiting, and more. The current fingerprinted Pupputeer scraper fails because it doesn't cover all these detection measures.

Fortunately, there's an easy solution to these limitations. You'll find out in the next section.

The Best Scraping Techniques to Avoid Detection

You can reduce the likelihood of detection by mimicking a real browser environment with web scraping request headers. Puppeteer also supports proxy, which allows you to mask requests behind different IPs. You can even apply manual Puppeteer patching to improve stealth.

However, these techniques are insufficient at scale, especially when dealing with advanced anti-bot measures.

Bypassing anti-bots during scraping requires thoroughly removing all bot-like signals. It involves implementing a solution that efficiently handles active blocks like CAPTCHAs and other challenges. The easiest way to achieve this is to use a web scraping solution like the ZenRows' Universal Scraper API.

The ZenRows' Universal Scraper API is efficient and reliable for avoiding even the most advanced anti-scraping mechanisms at scale. It's an all-in-one tool that uses cutting-edge technology under the hood to bypass fingerprinting and other detection mechanisms. 

With a single API call, ZenRows provides consistent request header optimization, premium proxy rotation, JavaScript rendering support, anti-bot and CAPTCHA auto-bypass, and more. It also has headless browser features, making it an excellent replacement for Puppeteer's automation.

Let's see ZenRows in action by scraping the Anti-bot Challenge page that previously blocked your scraper.

Sign up and go to the ZenRows Request Builder. Paste the target URL in the link box and activate Premium Proxies and JS Rendering.

Then, choose Node.js as your programming language and select the API connection mode. Copy and paste the generated code into your crawler file:

The generated code looks like this:

// npm install axios
const axios = require('axios');

const url = 'https://www.scrapingcourse.com/antibot-challenge';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
    url: 'https://api.zenrows.com/v1/',
    method: 'GET',
    params: {
        url: url,
        apikey: apikey,
        js_render: 'true',
        premium_proxy: 'true',
    },
})
    .then((response) => console.log(response.data))
    .catch((error) => console.log(error));

The code accesses the protected website and outputs its full-page HTML:

<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉 You just used ZenRows to bypass an anti-bot measure.

Conclusion

You've seen how Puppeteer fingerprinting can facilitate anti-bot detection. You've also learned how to use the puppeteer-with-fingerprints plugin to modify browser fingerprints to reduce the chances of detection.

While browser fingerprinting spoofing provides a solid footing to avoid blocks during scraping, it's often insufficient. We recommend using a scraping solution like ZenRows to handle all sorts of anti-bot measures and scrape without limitations. ZenRows is your one-stop solution for everything scraping and anti-bot bypass.

Try ZenRows for free today!