Device fingerprinting is one of the hidden but major blockers of web scraping. If you continue to get blocked, some details in your device configuration may be exposing you. But no worries!
This article explains how device fingerprinting works and shows you the best way to bypass it during scraping.
What Is Device Fingerprinting?
Device fingerprinting is a technique used to identify devices connecting to a server or other resource. It collects information about the device, including brand name, IP address, time zone, operating system, screen resolution, etc.
Any device can be fingerprinted as long as it connects to the internet. So, if you use a PC, a phone, or even a smart television, chances are you're being fingerprinted.
That said, device fingerprinting plays an important role in web scraping because anti-bots use it to profile devices to distinguish between human and automated request sources.
So, how exactly does device fingerprinting work? Let's dig deeper in the next section.
How Does Device Fingerprinting Work?
Although devices share many similar attributes, small details distinguish them. Even those belonging to the same brand and product line maintain some unnoticeable differences.
During device fingerprinting, this slight configuration difference between devices is enough information for web servers and anti-bot to profile their uniqueness.
Devices have many data points, and it's hard to tell which is used as the unique identifier at any given time. You might be surprised by how much insight the differences in battery levels, installed plugins and applications, IMEI number, motherboard serial number, MAC address, and system UUID can reveal to an anti-bot or a fingerprinting server.
How do all these block you during web scraping?
Anti-bot measures usually maintain a database of known bot device fingerprints. If you've made an automated request from your device in the past, you've likely been added to such a blocklist.
When you make a scraping request, anti-bots not only collect information about your HTTP client or browser, but also gather details about your device.
It's actually hard to determine a bot based on its device configuration. However, a combination of other request factors can be tagged with the device fingerprints and used as a yardstick to determine its bot-like nature. These include browser, HTTP, canvas fingerprints, headers, screen resolution, and more.
For instance, a non-fortified headless browser request without a GUI can reveal to an anti-bot that you're a bot. This way, the anti-bot can collect unique information about the source device and block subsequent requests.
This underlying mechanism of connecting the dots between the device information and other fingerprinting techniques makes device fingerprinting challenging for scrapers to bypass.
What Information Is Collected During Device Fingerprinting?
The device fingerprinting technique collects plenty of information, including hardware-level, browser-specific, and request-level data. Each information level is converted into specific hashes to encrypt each device's information and uniquely identify it more accurately.
The information collected during device fingerprinting can be grouped into the following hashes:
- Browser or HTTP Client Hash: These include browser details, such as navigator fields, request headers, fonts, time zone, plugins, IP address, browser language, screen resolution, color depth, and canvas information.
- Storage-Based Hash: The storage hash is generated per browser session and includes cookies, cookie IDs, and LocalStorage data.
- Hardware Hash: The encrypted data includes device type, operating system version, UUID, battery level, local time zone, language, keyboard configurations, touch support, Bluetooth and USB device information, WebRTC data, RAM and hard drive sizes, and the number of CPU cores.
Device Fingerprinting vs. Browser Fingerprinting
Device and browser fingerprinting sometimes collect similar data points, and you can actually confuse them as the same. Despite this close relationship, they still differ in intent and functionality.
Browser fingerprinting focuses on data related to the browser environment. Its goal is to identify the instance of the browser making a request at a particular period.
Device fingerprinting, on the other hand, is device-wide and aims to detect the unique characteristics of a device. The tracked data points cover hardware and software-level information, including the browser data itself.
Overall, keep in mind that while browser fingerprinting is limited to a single browser, device fingerprinting may persist across multiple browsers on the fingerprinted device.
Let's see how to bypass device fingerprinting in the next section.
How to Bypass Device Fingerprinting
Device fingerprinting can be a recurring blocker during web scraping if you don't handle it properly.
Solutions like proxies and User Agent spoofing often fail and are unsustainable, as this technique profiles many aspects of a device. And worse, the fingerprinted parts are unpredictable, so spoofing is less effective. That said, the solution below works every time.
Bypass Device Fingerprinting With a Web Scraping API
The easiest way to bypass device fingerprinting and scrape without getting blocked is via a web scraping solution like the ZenRows Universal Scraper API. The tool helps you evade all anti-bot mechanisms behind the scenes, so you don't have to worry about time-consuming and resource-intensive custom solutions ever again. Its proxy rotation and geo-targeting features also come through when accessing geo-locked content.
In addition to bypassing anti-bot measures, ZenRows' Universal Scraper API has headless browser features to handle dynamic rendering, ensuring you don't miss important data.
Let's quickly see how ZenRows' Universal Scraper API works by scraping the full-page HTML of this Antibot Challenge page, a heavily protected website.
Sign up and go to the Request Builder. Then, paste the target URL in the link box and activate Premium Proxies and JS Rendering.
Select your preferred programming language (in this case, Python) and choose the API connection mode. Copy the generated code and paste it into your scraper.
The generated Python code looks like this:
# pip3 install requests
import requests
url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
The above request bypasses the anti-bot challenge and outputs the full-page HTML of the protected site. See the result below:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! You just used ZenRows to bypass an anti-bot measure. You're now better equipped to scrape any website, regardless of its security level.
Other Methods
- Use Stealth Plugins: Stealth plugins, such as Puppeteer Stealth and SeleniumBase with Undetected ChromeDriver, have built-in evasion mechanisms that can help you bypass detection. The only limitation is that they eventually get blocked, as anti-bot measures continually upgrade to clamp down on open-source solutions. Additionally, these tools also exhibit bot-like attributes, leaving traces of automation.
- Disable JavaScript Execution: Since most device fingerprinting techniques rely on JavaScript execution, disabling JavaScript can prevent it completely. That said, this method is only recommended for static websites. Using it with dynamically rendered sites can prevent the target content from loading, resulting in empty data. Regardless, you should be careful, as disabling JavaScript can break major website functionalities, even for static websites.
Conclusion
Device fingerprinting is one of many anti-bot techniques that can block web scraping if not adequately handled. You've learned about what it is, how it works, and most importantly, how to bypass it.
While other bypass mechanisms might help, they're unsustainable, unscalable, costly, time-consuming, and technically draining. We recommend using a web scraping solution like ZenRows to handle all these tedious tasks, allowing you to focus on obtaining the data that matters to you. It's the one-stop tool to consider if you want to scrape at scale without limitations.