How to Bypass Cloudflare in Golang

Is your Golang scraper getting blocked by Cloudflare? Cloudflare's protective measures are a tough nut to crack. But you can bypass Cloudflare in Golang with the right tools.

You'll learn three of the most reliable methods to do so in this guide. But first, let's see more about what we're up against.

How Does Cloudflare Detect Golang's Bots?

Cloudflare is a content delivery network (CDN) providing websites with internet security and performance optimization services, similar to competitors like Akamai and DataDome. It reroutes users' traffic through its global network of servers and uses diverse active and passive techniques to detect bots, including:

IP Reputation and Request Rate Analysis 

Cloudflare's detection system profiles incoming IP traffic by comparing addresses against databases of trusted and suspicious IPs. If the IP matches a suspicious entry in the suspicious list, Cloudflare may flag it as bot-like and impose rate limiting or other challenges.

Concurrency model can also be risky for web scraping in Golang if not managed carefully. Sending multiple requests rapidly can overwhelm the server or trigger rate limiting, resulting in IP bans. 

Behavioral Analysis

Cloudflare also monitors regular user behavior, such as clicking, scrolling, navigation, typing, etc. While human user interactions are typically unpredictable, Golang's scraping libraries often maintain a fixed pattern per request.

Fixed behavioral patterns can flag you as a bot, such as clicking the same link repeatedly, scrolling the same height at regular intervals, completing a form within milliseconds, etc.

CAPTCHA

Cloudflare occasionally uses its Turnstile CAPTCHA box to confirm if you're an authentic user. Cloudflare Turnstile CAPTCHA requires two rounds of authentication on a single interstitial page.

First, the user's browser must complete a JavaScript challenge within a particular period. If this challenge is unsuccessful or causes suspicion, the user could be required to click a CAPTCHA checkbox to verify legitimacy. Cloudflare will flag and block your scraper if it doesn't finish these tasks.

HTTP Request Header Analysis

Your HTTP request headers are a crucial aspect of your scraper. Cloudflare also analyzes the request headers to determine if a request is legitimate.

Inconsistencies like mismatches or missing header information can lead to detection. For instance, pairing a Linux Platform header with a Windows Chrome User Agent may result in detection and subsequent blocking.

Canvas Fingerprinting

Cloudflare also uses Canvas fingerprinting as a detection technique by leveraging the HTML5 Canvas element's ability to render hidden images or texts within the user's browser. The rendering varies with the browser's operating system, version, and device graphic card.

Cloudflare compares this fingerprint to known patterns, detecting discrepancies that might signal bot-like behavior, as many automated tools do not replicate these rendering details accurately.

By using a combination of these and other techniques, Cloudflare can detect and mitigate bot traffic. However, there are ways to bypass its measures in Go and access the information you want. Let's see three effective techniques.

Method #1: Using Playwright for Go

Playwright is a browser automation library that simulates a browser environment efficiently and supports the DevTools protocol. With this ability, you can mimic human-like interactions like scrolling, clicking, filling out forms, and more. However, while this feature may increase your chances of bypassing Cloudflare security measures, it's often insufficient.

Playwright and other standard automation libraries leak bot-like attributes, such as the presence of a WebDriver, missing plugins, etc, making them easily detectable. Unfortunately, Playwright doesn't support stealth plugins for Go, but you can check out our guide on bypassing Cloudflare with Playwright to learn how to optimize your Scraper.

Method #2: Use a Web Scraping API

The most reliable way to bypass Cloudflare is to use a web scraping API, such as the ZenRows' Universal Scraper API. It provides an all-in-one toolkit for bypassing Cloudflare without limitations.

The ZenRows Scraper API handles JavaScript rendering, proxy management, actual user spoofing, fingerprinting evasion, request header management, and all other anti-bot and CAPTCHA bypass mechanisms. It's highly scalable and only requires sending a single API call, making it the easiest option, especially for beginners.

Let's quickly see how it works by scraping the full-page HTML of this Cloudflare challenge page.

Sign up to open the ZenRows Request Builder. Paste the target URL in the link box and activate Premium Proxies and JS Rendering.

Select Go as your programming language and choose the API connection mode. Copy and paste the generated code into your Golang scraper file.

The generated code should look like this:

package main

import (
    "io"
    "log"
    "net/http"
)

func main() {
    client := &http.Client{}
    req, err := http.NewRequest("GET", "https://api.zenrows.com/v1/?apikey=<YOUR_ZENROWS_API_KEY>&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fcloudflare-challenge&js_render=true&premium_proxy=true", nil)
    resp, err := client.Do(req)
    if err != nil {
        log.Fatalln(err)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatalln(err)
    }

    log.Println(string(body))
}

The code outputs the protected website's full-page HTML as shown:

<html lang="en">
<head>
    <!-- ... -->
    <title>Cloudflare Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Cloudflare challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

That works 🎉! You just scraped a Cloudflare-protected website using ZenRows.

Method #3: go-rod/Stealth

Rod (go-rod/rod) is a customizable browser automation and web scraping WebDriver based on the DevTools Protocol. It also allows you to mimic user interactions like filling out forms, clicking, or extracting the value of input elements by class. However, it presents bot-like properties that make it detectable.

Pairing Rod with the go-rod/stealth plugin adds extra stealth functionality. The go-rod/stealth plugin features various evasion strategies to patch the bot-like attributes of the standard Rod library. Still, even the stealth plugin doesn't guarantee you'll bypass Cloudflare in Golang at scale.

You can combine them with other measures like web scraping proxies for better results.

Conclusion

As you can see, Cloudflare has set numerous traps for your scraper, and you need to be ready for the challenge. Sometimes, you can rely on tools like Playwright or go-rod with go-rod/stealth to extract data without being detected.

However, they'll inevitably fall short, particularly when scaling scraping operations or dealing with advanced Cloudflare protections. The best solution to bypass anti-bots at scale is to integrate a web scraping API such as ZenRows, which provides all the tools you need for the job. 

Try ZenRows for free now without a credit card!