How to Integrate Puppeteer with ZenRows

const browser = await puppeteer.launch({
    ignoreHTTPSErrors: true,
});

Puppeteer will now be able to visit any HTTPS page with no issues.

Error Solving: Network Failed and Empty Response Errors

There are a few ways to integrate proxies into Puppeteer, but not all of them work with ZenRows. The page.authenticate() method will produce the net::ERR_EMPTY_RESPONSE error. Similarly, using the puppeteer-page-proxy package will result in net::ERR_FAILED.

As of this writing, the proxy-chain approach presented earlier is the only way to make ZenRows work with Puppeteer.

Error Solving: Limited Concurrent Requests

On a free trial, ZenRows supports up to 5 simultaneous requests with no queuing or auto-retry. When the target page involves a lot of resources, that can become a problem.

The reason is that the headless browser will make many concurrent requests, and some of them will fail with a 429 Too Many Requests error.

There are two ways to tackle this issue: A) Configure Puppeteer to avoid loading unnecessary resources. B) Upgrade to a ZenRows paid plan to gain access to a higher level of concurrency.

FAQ: What Happens When You Enable ZenRows‘s JavaScript Rendering with Puppeteer? If you select the “JS Rendering” option when building the ZenRows proxy URL to integrate into Puppeteer, two things will happen:

• ZenRows and Puppeteer will both render the page: ZenRows will render the page in a headless browser and, if set, run the specified JS instructions. Puppeteer will then receive the HMTL returned by ZenRows and render it again. This process only introduces unnecessary overhead and brings no real benefit.
• You'll lose the interaction session: The original session ends when ZenRows returns the page HTML. If you instruct Puppeteer to interact with the page, a new session will start. This may activate anti-bot systems, since the simulated interaction with the page isn't consistent.

In short, the double rendering setup makes no sense because of performance and effectiveness reasons. Therefore, don't activate ZenRows' js_render parameter together with Puppeteer.

• Stopped by Bot Detection with Puppeteer: CAPTCHAs, WAFs, and Beyond Websites know how valuable their data is, and that's why many of them protect their pages with advanced anti-bot measures. These technologies can detect and block automated requests, preventing scripts like yours from accessing their pages.

Proxies are usually not enough to get around these anti-scraping solutions. Some of these systems, such as the WAF used by G2.com, even rely on machine learning!

As you can imagine, implementing custom logic and workarounds in Puppeteer isn't effective either.

The solution? Replacing Puppeteer with ZenRows!

ZenRows provides the same JavaScript rendering and user interaction simulation capabilities as Puppeteer. On top of that, it provides the most complete anti-bot bypass toolkit available. As you're about to see in the following section, that makes it the ideal Puppeteer alternative.

Option 2: Replace Puppeteer with ZenRows to Stop Fearing Anti-Bots

ZenRows has the same functionality than Puppeteer, but guarantees greater effectiveness against anti-bot measures, and also reduces maintenance and machine costs.

It's time to explore how to replace Puppeteer with ZenRows and verify it can scrape any web page on the web!

Step 1: Get Your ZenRows Account

Sign up for ZenRows to retrieve your API key and 1,000 free API credits, or log in.

Step 2: Set Up Your Request

You'll get to the Request Builder. There, paste your target URL (e.g., https://www.g2.com/products/jira/reviews), enable the AI Anti-bot mode (which also comes with JS rendering), and check the Premium Proxies option.

Then, select your programming language (if it isn't available in the builder, select "cURL" to get the full endpoint you can call with any HTTP client).

Lastly, choose the "API" connection mode (when available, "SDK" is also a viable option).

If you want to perform some interaction on the page, check "JavaScript Instructions" box. Take a look at all JS actions available.

Let's see how to scrape data from a page with the API mode.

Step 3: Get the Unlocked HTML Via API

Still the Request Builder, click the "Try it" button to test the configured API endpoint directly. Once you've verified it works, you're ready to call it in your script.

Now, retrieve your Node.js snippet by clicking the "Copy code" button (but remember that ZenRows works with any programming language). Paste it into your scraper.js script:

// scraper.js

// npm install axios
const axios = require("axios");

const url = "https://www.g2.com/products/asana/reviews";
const apikey = "<YOUR_ZENROWS_API_KEY>";
axios({
    url: "https://api.zenrows.com/v1/",
    method: "GET",
    params: {
        "url": url,
        "apikey": "<YOUR_ZENROWS_API_KEY>",
        "js_render": "true",
        "antibot": "true",
        "premium_proxy": "true",
        // 'js_instructions': your JS instructions...,
    },
})
    .then(response => console.log(response.data))
    .catch(error => console.log(error));

Next, install the axios library (or use the HTTP client package you prefer):

node scraper.js

<!DOCTYPE html>
<head>
  <meta charset="utf-8" />
  <link href="https://www.g2.com/assets/favicon-fdacc4208a68e8ae57a80bf869d155829f2400fa7dd128b9c9e60f07795c4915.ico" rel="shortcut icon" type="image/x-icon" />
  <title>Jira Reviews 2023: Details, Pricing, &amp; Features | G2</title>
  <!-- omitted for brevity ... -->