The Anti-bot Solution to Scrape Everything? Get Your Free API Key! ๐Ÿ˜Ž

How to Bypass Cloudflare in C#

February 2, 2023 ยท 5 min read

Can you bypass Cloudflare in C#? Since you're here asking this question, we can take a wild guess that you've tried scraping a webpage protected by Cloudflare, and it didn't work.

The best way to solve this challenge is to use techniques capable of masking your crawler and slipping it through the Cloudflare bot detection radar.ย 

In this article, we'll show you how to do a C# Cloudflare bypass. Let's get started!

What Is Cloudflare?

Cloudflare is a company that provides a service to help protect and accelerate websites. Its data centers network acts as a reverse proxy for websites. So when users visit a protected website, their request is routed through Cloudflare's network before it reaches the origin server.

That allows Cloudflare to provide services such as DDoS protection, content optimization, and SSL/TLS encryption.

A few Bot Management errors you might run into trying to scrape a web page without bypassing Cloudflare in C# include:

  • Error 1010.
  • Error 1012.
  • Error 1020.

They are followed by a 403 Forbidden HTTP response status code.

Can Cloudflare Detect C#?

Yes, Cloudflare is capable of detecting C# scrapers and blocking them. To confirm this, let's use a basic code to scrape OpenSea:

Start by opening a visual studio instance and create a console application. Then execute the following code:

scraper.cs
WebClient wc = new WebClient(); 
var webStream = wc.OpenRead("https://opensea.io/category/art"); 
var webReader = new StreamReader(webStream); 
var content = webReader.ReadToEnd();

Here's what the result will look like:

Naive doesn't work
Click to open the image in full screen

The output shows a request was sent to https://opensea.io/category/art and Cloudflare returned a Cloudflare 403 Forbidden Error. So how can you bypass C# Cloudflare and scrape without headaches? Let's get into that.

Frustrated that your web scrapers are blocked once and again?
ZenRows API handles rotating proxies and headless browsers for you.
Try for FREE

How to Bypass Cloudflare in C#?

There are many different methods to bypass Cloudflare in C# using libraries. We'll discuss the ones we found effective using the same target website employed above since it uses Cloudflare.ย 

Our mission will be to scrape the names of Opensea's trending collection from the art category.

ZenRows

The easiest and most effective way to bypass C# Cloudflare is using ZenRows, a web scraping API that bypasses Cloudflare and all major anti-bots.

๐Ÿ‘ Pros:

  • It's easy to use.
  • No installation is required.
  • It integrates seamlessly with C#.
  • You can scrape dynamic web pages.

๐Ÿ‘Ž Cons:

  • It's a paid service, yet offers a free trial, and plans start as low as $49/month.

How to Bypass Cloudflare in C# Using ZenRows?

To do C# Cloudflare bypass using ZenRows, start by creating a free account to access the Request Builder. Then, paste your target URL and check the boxes for premium proxy and anti-bot. That will give you the API key and parameters to use in our scraper:

ZenRows Proxy Dashboard
Click to open the image in full screen

To call ZenRows' API from C#, add the following ZenRowsCFBypass function to the Program.cs file:

scraper.cs
public static async Task<string> ZenRowsCFBypass(string url) 
{ 
	var client = new HttpClient(); 
	var encodedURL = WebUtility.UrlEncode(url); 
	var apiKey = "YOUR_API_KEY_HERE"; 
 
	var parameters = "premium_proxy=true&antibot=true"; 
	var request = $"https://api.zenrows.com/v1/?apikey={apiKey}&url={encodedURL}&{parameters}"; 
	var result = await client.GetStringAsync(request); 
	return result; 
}

Run the script by adding the following code in the Main method:

scraper.cs
var rawHTML = ZenRowsCFBypass("https://opensea.io/category/art").Result;

The scraper will get the entire HTML of the protected page in the rawHTML variable, and the output should look like this:

Output Proxy Sharp
Click to open the image in full screen

Congrats on successfully bypassing Cloudflare in C# using ZenRows!

Here's what the complete code looks like:

scraper.cs
using System.Net; 
using System.Net.Http; 
 
public class ZenRowExample 
{ 
	public static void Main(string[] args) 
	{ 
		var rawHTML = ZenRowsCFBypass("https://opensea.io/category/art").Result; 
	} 
 
	public static async Task<string> ZenRowsCFBypass(string url) 
	{ 
		var client = new HttpClient(); 
		var encodedURL = WebUtility.UrlEncode(url); 
		var apiKey = "YOUR_ZENROWS_API_KEY"; 
		var parameters = "premium_proxy=true&antibot=true"; 
		var request = $"https://api.zenrows.com/v1/?apikey={apiKey}&url={encodedURL}&{parameters}"; 
		var result = await client.GetStringAsync(request); 
	 
		return result; 
	} 
}

Puppeteer-Sharp

Puppeteer-Sharp is another method used in C# to bypass Cloudflare. It's a NetStandard 2.0 library, and the minimum platform versions are .NET Framework 4.6.1 and .NET Core 2.0.

๐Ÿ‘ Pros:

  • It's easy to use.
  • It can scrape dynamic web pages.

๐Ÿ‘Ž Cons:

  • It uses a lot of memory and CPU resources.

How to Bypass Cloudflare in C# Using Puppeteer-Sharp?

To bypass Cloudflare in C# using Puppeteer-Sharp, create a new Console App project in Visual Studio and name it OpenSeaPupSharp. Then, add the following starting code to Program.cs:

scraper.cs
public class PupSharp 
{ 
	public static void Main(string[] args) 
	{ 
		Console.WriteLine("Trending arts..."); 
	} 
}

The next step is to get the library from Nuget, which can be done by navigating to Tools and then Nuget Package Manager, as shown below.

Package Manager
Click to open the image in full screen

From the Nuget Package Manager, install the Puppeteer-Sharp library. After installation, it'll show under the dependency for the project:

Terminal
Install-Package PuppeteerSharp -Version 8.0.0

Now, go to the Main method and write the following code:

scraper.cs
using PuppeteerSharp; 
 
public class PupSharp 
{ 
	public static void Main(string[] args) 
	{ 
		CheckingHeadLessChrome().Wait(); 
	} 
 
	// Just to see PuppeteerSharp in action in an headless chrome settings 
	public static async Task CheckingHeadLessChrome() 
	{ 
		string outputFile = "Shot.png"; 
		using var browserFetcher = new BrowserFetcher(); 
		await browserFetcher.DownloadAsync(); 
		var options = new LaunchOptions() 
		{ 
			ExecutablePath = @"C:\Program Files\Google\Chrome\Application\chrome.exe", 
			Headless = true, 
			SlowMo = 10 
		}; 
		await using var browser = await Puppeteer.LaunchAsync(options); 
		await using var page = await browser.NewPageAsync(); 
		await page.GoToAsync("https://opensea.io/category/art"); 
		var allContent = await page.GetContentAsync(); 
		await page.ScreenshotAsync(outputFile); 
	} 
}

It creates a fake browser using browserFetcher. It downloads the entire HTML via browserFetcher.DownloadAsync(), and the SlowMo method is set to ten. That generates the screenshot below:

Blocked
Click to open the image in full screen

Oops, Cloudflare caught up to us! It detected the scraper and concluded that the request was coming from a probable malicious bot since the Headless flag is set to true. Set Headless = false to fix this and rerun the code.

scraper.cs
var options = new LaunchOptions() 
{ 
	ExecutablePath = @"C:\Program Files\Google\Chrome\Application\chrome.exe", 
	Headless = false, 
	SlowMo = 10 
};

With this change, the scraper wasn't blocked:

With Chrome
Click to open the image in full screen

CloudProxySharp

CloudProxySharp is a proxy server used to bypass Cloudflare protection. It works by waiting for user requests in an idle state using few resources. When some request arrives, it uses Puppeteer with the Stealth plugin to create a headless browser (Chrome).

๐Ÿ‘ Pros:

  • Easy to use.
  • Faster than some other libraries.

๐Ÿ‘Ž Cons:

  • It's unstable since it's still in its beta phase.

How to Bypass Cloudflare in C# Using CloudProxySharp?

Start by installing the Nuget package and then install CloudProxySharp using the command below:

Terminal
Install-Package CloudProxySharp -Version 1.0.2

After installing the library, create the CloudProxySharpCFBypass function. It wraps the functionality to get the text from the scraped page.

scraper.cs
public static Task<string> CloudProxySharpCFBypass(string url) 
{ 
	var handler = new ClearanceHandler("http://localhost:8191/") 
	{ 
		//Make sure that the string literal is in single line else it won't work 
		UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36", 
		MaxTimeout = 60000 
	}; 
 
	var client = new HttpClient(handler); 
	var content = client.GetStringAsync(url); 
	return content; 
}

The function creates an HttpClient that fakes a real browser. Then, this client is used to get the raw HTML from the web page we're trying to scrape. The handler of the HttpClient is configured to wait up to one minute by setting the MaxTimeout property. The process of downloading the raw HTML is asynchronous.

You can now go ahead and call the function:

scraper.cs
public static void Main(string[] args) 
{ 
	var content = CloudProxySharpCFBypass("https://opensea.io/category/art").Result; 
}

And there you have it! Here's what the output looks like:

Output Proxy Sharp
Click to open the image in full screen

The screenshot shows the debug view of the raw HTML captured.

Conclusion

Cloudflare Bot Management is one of the headaches of C# web scraping as it can detect and block scrapers. This article discussed three C# Cloudflare bypass libraries.ย 

However, they become unreliable when scaling and making more requests, except for ZenRows. It easily integrates smoothly with C#, and you can get your free API key.

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn.

Frustrated that your web scrapers are blocked once and again? ZenRows API handles rotating proxies and headless browsers for you.
Try for FREE

The easiest way to do Web Scraping

From Rotating Proxies and Headless Browsers to CAPTCHAs, a single API call to ZenRows handles all anti-bot bypass for you.