Puppeteer's browser automation feature follows a regular bot-like pattern, making it prone to anti-bot detection. The puppeteer-humanize library can improve your scraper's stealth by adding a human touch to your keystrokes during scraping.
We'll show you how puppeteer-humanize works and how to use it for scraping in Puppeteer.
What Is puppeteer-humanize?
puppeteer-humanize is a Node.js library that allows you to mimic human typing behaviors when automating form filling with Puppeteer.
Anti-bots often analyze a user's behavior as part of their detection mechanisms. Since human interactions with web elements are dynamic, unrealistic or overly coordinated actions are obvious bot-like signals for anti-bots.
The standard Puppeteer library executes unrealistic bot-like behaviors, making it detectable during scraping. These include:
- Scrolling the same height several times at the same interval.
- Clicking the same element at the same pace.
- Typing more rapidly or smoothly than expected without mistakes.
- A stationary cursor or one that moves in a straight line or a single direction.
- Zero reaction to layout shifts or dynamic events, such as alerts or modal popups.
puppeteer-humanize focuses on the typing aspect of automation to add the following human touch to your Puppeteer scraper:
- The library allows you to simulate typographical errors in input fields.
- It mimics the human backspace keystroke to erase simulated mistakes.
puppeteer-humanizeretypes deleted text inputs using random delays.- It adds random delays between keystrokes to humanize typing speed.
The library works directly with base Puppeteer, but you can pair it with Puppeteer Extra to further increase stealth behavior.
However, as mentioned, puppeteer-humanize is specifically suitable for humanizing form inputs and isn't entirely an anti-bot bypass tool. While it makes your Puppeteer scraper less detectable, it doesn't guarantee you'll escape anti-bots.
You'll learn how puppeteer-humanize works in the next section.
How to Scrape With puppeteer-humanize
To learn how puppeteer-humanize works, you'll use it to simulate typing on the Cloudflare-protected Login Challenge form. Here's what the page looks like when the anti-bot challenge is solved:
After filling in the input fields, you'll screenshot the page to see if puppeteer-humanize passes the anti-bot challenge.
Install the Plugin
First, install puppeteer-humanize, Puppeteer Extra, and puppeteer-extra-plugin-stealth:
npm install @forad/puppeteer-humanize puppeteer-extra puppeteer-extra-plugin-stealth
Now, let's see the tool in action.
Scrape Data Using puppeteer-humanize
To scrape with puppeteer-humanize, import it along with the Puppeteer Extra libraries and activate the stealth plugin:
// npm install @forad/puppeteer-humanize puppeteer-extra puppeteer-extra-plugin-stealth
const { typeInto } = require('@forad/puppeteer-humanize');
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
// add the stealth plugin
puppeteer.use(StealthPlugin());
Launch the browser instance in headless mode, open the target page, and get the input selectors:
// ...
(async () => {
try {
// launch a browser instance in headless mode
const browser = await puppeteer.launch();
const page = await browser.newPage();
// open the target page
await page.goto('https://www.scrapingcourse.com/login/cf-turnstile');
// get the input selectors
const emailInput = await page.$('#email');
const passwordInput = await page.$('#password');
const submitElement = await page.$('#submit-button');
await browser.close();
} catch (error) {
console.error('Error:', error);
}
})();
Next, configure puppeteer-humanize to customize human behaviors such as typing mistakes, random delays, and controlled typing speed. Input the texts using the typeInto function from puppeteer-humanize and screenshot the page to capture the anti-bot challenge action:
// ...
(async () => {
try {
// ...
// configure puppeteer-humanize to simulate human typing behavior
if (emailInput && passwordInput && submitElement) {
const config = {
mistakes: {
chance: 8,
delay: {
min: 50,
max: 500,
},
},
delays: {
space: {
chance: 70,
min: 10,
max: 50,
},
},
};
// type into the fields
await typeInto(emailInput, '[email protected]', config);
await typeInto(passwordInput, 'password', config);
new Promise((resolve) => setTimeout(resolve, 15000));
// take a screenshot after typing
await page.screenshot({ path: 'screenshot.png' });
} else {
console.log('Missing email, password, or submit button field');
}
// ...
} catch (error) {
// ...
}
})();
Combine the snippets, and you'll get the following complete code:
// npm install @forad/puppeteer-humanize puppeteer-extra puppeteer-extra-plugin-stealth
const { typeInto } = require('@forad/puppeteer-humanize');
const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
// add the stealth plugin
puppeteer.use(StealthPlugin());
(async () => {
try {
// launch a browser instance in headless mode
const browser = await puppeteer.launch();
const page = await browser.newPage();
// open the target page
await page.goto('https://www.scrapingcourse.com/login/cf-turnstile');
// get the input selectors
const emailInput = await page.$('#email');
const passwordInput = await page.$('#password');
const submitElement = await page.$('#submit-button');
// configure puppeteer-humanize to simulate human typing behavior
if (emailInput && passwordInput && submitElement) {
const config = {
mistakes: {
chance: 8,
delay: {
min: 50,
max: 500,
},
},
delays: {
space: {
chance: 70,
min: 10,
max: 50,
},
},
};
// type into the fields
await typeInto(emailInput, '[email protected]', config);
await typeInto(passwordInput, 'password', config);
new Promise((resolve) => setTimeout(resolve, 15000));
// take a screenshot after typing
await page.screenshot({ path: 'input-screenshot.png' });
} else {
console.log('Missing email, password, or submit button field');
}
await browser.close();
} catch (error) {
console.error('Error:', error);
}
})();
The above code outputs the following screenshot. Unfortunately, the anti-bot blocked puppeteer-humanize, as shown below:
The puppeteer-humanize setup got blocked despite simulating human typing actions and pairing it with the Puppeteer stealth plugin.
While puppeteer-humanize increases stealth through human actions, it has several limitations that make it ineffective against anti-bot systems. You'll learn more about them in the next section, including the best solution to bypass these blocks.
Limitations of puppeteer-humanize and the Best Alternative
There are significant limitations to using puppeteer-humanize. First, the library hasn't received an update in 4 years. Since anti-bot measures frequently improve their security and clamp down on open-source tools, this lack of updates makes puppeteer-humanize readily susceptible to blocking.
Additionally, merely simulating human interactions doesn't shield your scraper from detection, but only increases the chances of evading it. Anti-bot measures combine behavioral analysis with other anti-scraping measures. These include browser fingerprinting, JavaScript challenges, CAPTCHAs, IP analysis, and more. You won't bypass these other detection mechanisms even if you evade behavioral checks. This eventually results in blocking.
The most reliable way to bypass any anti-bot measure at scale is to use a web scraping solution, such as the ZenRows Universal Scraper API. ZenRows handles human interaction simulations and all other anti-bot bypass technicalities behind the scenes. It lets you focus on core business decisions rather than worry about anti-scraping measures.
ZenRows also supports JavaScript rendering and headless browsing, making it a suitable replacement for Puppeteer. A single API call is all you need to use these features.
Let's quickly see how the ZenRows Universal Scraper API works by scraping a heavily protected site like the Antibot Challenge page.
Sign up and go to the Request Builder. Then, paste the target URL in the link box and activate Premium Proxies and JS Rendering.
Select Node.js as your preferred programming language and choose the API connection mode. Copy the generated code and paste it into your scraper.
The generated JavaScript looks like this:
// npm install axios
const axios = require('axios');
const url = 'https://www.scrapingcourse.com/antibot-challenge';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
url: 'https://api.zenrows.com/v1/',
method: 'GET',
params: {
url: url,
apikey: apikey,
js_render: 'true',
premium_proxy: 'true',
},
})
.then((response) => console.log(response.data))
.catch((error) => console.log(error));
The above request bypasses the anti-bot challenge and outputs the full-page HTML of the protected site. See the result below:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! 🎉 You just used ZenRows to bypass the anti-bot measure. Your scraper can now access any website without limitations.
Conclusion
You've learned to humanize your Puppeteer scraper using puppeteer-humanize. Despite the library's ability to add human behavior to Puppeteer, it doesn't work against advanced anti-bot measures.
ZenRows is the best solution for bypassing any anti-bot measure, regardless of complexity. It's an all-in-one toolkit to scrape data on a large scale without getting blocked.