Cloudflare represents one of the biggest challenges for web scraping, as it can detect and block most automated requests. There are some methods to bypass Cloudflare, but ZenRows is the easiest and most reliable one.

Getting around Cloudflare’s security measures boils down to making a call to the ZenRows API. This approach works with any programming language and HTTP client library.

Let’s now look at the step-by-step procedure to get around Cloudflare with ZenRows.

TL;DR: Get Around Cloudflare with ZenRows API:

  1. Sign up for ZenRows to get your free API key.
  2. In the Request Builder, paste your target URL and activate the Premium Proxy and JavaScript Rendering parameters.
  3. Copy the generated code and integrate it into your code.

​
Step-by-Step: Use ZenRows to Skip Cloudflare

Suppose you want to retrieve data from a reviews page for Asana on G2.com, a site under Cloudflare:

https://www.g2.com/products/asana/reviews

If you request access to that web page in your scraping script, you’ll likely get a 403 Forbidden error.

Cloudflare’s anti-bot protection technology will intercept your request and stop it, and the same behavior occurs with all the other sites it protects. Remark that different sites and even different pages might have a different security level implemented.

Follow the steps below to avoid being blocked.

​
Step 1: Sign Up for ZenRows

Create a ZenRows account. You’ll get your API key and 1,000 free credits right away. You’ll also gain access to a support chat by and for developers, even on the free trial.

If you already have an account, use your credentials to log in.

​
Step 2: Set Up Your API Call

You’ll get to the Request Builder page.

ZenRows Builder Page

Here, do as follows:

  1. Paste your target URL (https://www.g2.com/products/asana/reviews).
  2. Activate the parameters Premium Proxy and JavaScript Rendering to allow you to bypass even CloudFlare’s highest security measures. Additionally, it’s recommended to use the “Wait For” option with a CSS Selector.
  3. Choose your programming language. If it isn’t in the list, select “cURL” to get the full API endpoint that you can call with any HTTP client.
  4. Select the “API” mode because it’s ideal for developers looking to test and use ZenRows in any programming language. Other connection modes will also work.

With that, you’re ready to win over Cloudflare with ZenRows. Head to the next step in this tutorial.

FYI About the different connection modes:

  • API: Endpoints you can call to get site data in your preferred language.
  • SDK: Libraries to integrate the ZenRows feature into an existing application.
  • Proxy: Servers you can configure your HTTP request to go through for anonymity and security.

​
Step 3: Scrape Using the API

You can test the API directly in the Request Builder by clicking the “Try it” button. Once you’re sure it works, go ahead to integrate it into your code.

Click “Copy to code” and retrieve the following Python code (but again, to overcome Cloudflare, ZenRows works with any programming language):

scraping.py
# pip install requests
import requests

url = 'https://www.g2.com/products/asana/reviews'
apikey = '<YOUR_ZENROWS_API_KEY>'
params = {
    'url': url,
    'apikey': apikey,
    'js_render': 'true',
    'wait_for': 'div.l2',
    'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)

Copy those lines into your Python script (e.g., scraping.py). Then, install the request library (any other HTTP request library will also work):

pip install requests

Now, run the script:

python scraping.py

It’ll print this output:

<!DOCTYPE html>
<head>
  <meta charset="utf-8" />
  <link href="https://www.g2.com/assets/favicon-fdacc4208a68e8ae57a80bf869d155829f2400fa7dd128b9c9e60f07795c4915.ico" rel="shortcut icon" type="image/x-icon" />
  <title>Asana Reviews 2023: Details, Pricing, &amp; Features | G2</title>
  <!-- omitted for brevity ... -->

That’s exactly the HTML content you were looking for!

Congrats! You just won over Cloudflare with ZenRows. 🎉

​
API Credit Pricing

Even different pages on the same site may have different levels of protection, as mention earlier. To cover all scenarios and maximize the success rate of your scraping requests, we recommend the three API parameters used above.

As you can see from the message in the Request Builder, that request will cost 25 credits.

API request cost

To be noted, credit use is based on successful requests only.

To overcome Cloudflare, ZenRows offers several plans, starting at $49 per month. For detailed information, check out our pricing page.

PuppeteerAPI Error Codes