How to Use a HttpClient Proxy in C# (2024)

HttpClient is one of the best libraries for making HTTP requests in .NET, enabling web scraping in C#. However, your automated requests may get identified as coming from a bot and blocked, so let's learn how to avoid that with a C# HttpClient proxy!

What Is an HttpClient Proxy

An HttpClient proxy acts as an intermediary between your app and the target server. Its purpose is to route requests made by the HTTP C# client through a different IP address so that the server will see them as coming from the proxy server instead of the original client.

This mechanism is useful to avoid getting blocked and bypassing geographical restrictions. Read on to find out how to implement it!

Prerequisites

If you don't have .NET installed on your machine, download the latest version of SDK. Double-click on the installer and follow the wizard.

To save time, you can install the .NET Coding Pack, including the .NET SDK, Visual Studio Code, and its .NET extensions.

Open PowerShell and verify that .NET works:

dotnet --list-sdks

It should print something like this:

7.0.5 [C:\Program Files\dotnet\sdk]

Next, follow the instructions below from the official guide to set up a C# project in Visual Studio Code:

1. Create a folder for your project:

mkdir HttpClientProxy

2. Open it in Visual Studio Code.
3. Initialize a .NET project with the command below in the VS Code terminal:

dotnet new console --framework net7.0 

The HttpClientProxy folder now contains a C# app. HttpClient is part of the C# system library, so you don't need to add it as an external package.

Use it to perform a basic HTTP request to HttpBin to get your IP by updating Program.cs as follows. The GetAsync() sends an HTTP GET request to the specified URL as an asynchronous operation. using makes sure the resources allocated for the request are released when no longer needed. Then, ReadAsStringAsync() gets the response returned by the server as a string.

using System.Net;

namespace HttpClientProxy
{
  class Program
  {  
    static async Task Main(string[] args)
    {
      // initialize an HttpClient instance
      HttpClient client = new HttpClient();

      try
      {
        // perform an async GET request to HttpBin
        using HttpResponseMessage response = await client.GetAsync("https://httpbin.org/ip");
        // extract the request response and print it
        string responseContent = await response.Content.ReadAsStringAsync();
        Console.WriteLine(responseContent);
      }
      catch (HttpRequestException e)
      {
        Console.WriteLine("Request failed with error: ", e.Message);
      }

    }
  }
}

This script will print your IP:

{
  "origin": "194.127.58.110"
}

Fantastic! You learned how to get started with HttpClient and are ready to use it with a proxy!

How to Use a Proxy with HttpClient in C#

First, get a valid proxy from Free Proxy List and store its URL in a string variable:

string proxyURL = "http://103.167.135.111:80"

As you can see, a free proxy URL follows the syntax below:

<PROXY_PROTOCOL>://<PROXY_IP_ADDRESS>:<PROXY_PORT>

By default, HttpClient reads the proxy configuration from the system settings. Change that behavior and set a proxy with a WebProxy instance like this:

// proxy configs
String proxyURL = "http://103.167.135.111:80";
WebProxy webProxy = new WebProxy(proxyURL);
// make the HttpClient instance use a proxy
// in its requests
HttpClientHandler httpClientHandler = new HttpClientHandler
{
  Proxy = webProxy
};
client = new HttpClient(httpClientHandler);

Here's what the entire code looks like:

using System.Net;

namespace HttpClientProxy
{
  class Program
  {
    static async Task Main(string[] args)
    {
      string proxyURL = "http://103.167.135.111:80";
      WebProxy webProxy = new WebProxy(proxyURL);

      HttpClientHandler httpClientHandler = new HttpClientHandler
      {
        Proxy = webProxy
      };
      HttpClient client = new HttpClient(httpClientHandler);

      try
      {
        using HttpResponseMessage response = await client.GetAsync("https://httpbin.org/ip");
        string responseContent = await response.Content.ReadAsStringAsync();
        Console.WriteLine(responseContent);
      }
      catch (HttpRequestException e)
      {
        Console.WriteLine("Request failed with error: ", e.Message);
      }
    }
  }
}

Launch the script, and you'll get the following output:

{
  "origin": "103.167.135.111"
}

That's the exact same IP of the proxy server, meaning the HttpClient is making requests through the specified proxy, as desired.

Perfect! You now know the basics of using a C# HttpClient proxy. It's time to explore more advanced concepts!

Proxy Authentication with HttpClient: Username & Password

Premium proxies protect their access through authentication. That way, only users with a valid pair of credentials can connect to their servers.

The URL of an authenticated proxy involves a username and a password. This is the usual syntax:

<PROXY_PROTOCOL>://<USERNAME>:<PASSWORD>@<PROXY_IP_ADDRESS>:<PROXY_PORT>

Yet, WebProxy doesn't support it and instead requires a Credentials instance:

WebProxy webProxy = new WebProxy
{
  // proxy URL with no credentials involved
  Address = new Uri("<PROXY_URL>"),
  // specify the proxy credentials
  Credentials = new NetworkCredential(
        userName: "<USERNAME>",
        password: "<PASSWORD>"
  )
};

Here's a complete example of how to use proxy authentication in HttpClient:

using System.Net;

namespace HttpClientProxy
{
  class Program
  {
    static async Task Main(string[] args)
    {
      // authenticated proxy info
      string proxyURL = "http://139.92.119.185:8080";
      string proxyUsername = "jiprkcdaumui";
      string proxyPassword = "tZqqUck4D5VSczwFU";

      WebProxy webProxy = new WebProxy
      {
        Address = new Uri(proxyURL),
        // specify the proxy credentials
        Credentials = new NetworkCredential(
              userName: proxyUsername,
              password: proxyPassword
        )
      };

      HttpClientHandler httpClientHandler = new HttpClientHandler
      {
        Proxy = webProxy
      };
      HttpClient client = new HttpClient(httpClientHandler);

      try
      {
        using HttpResponseMessage response = await client.GetAsync("https://httpbin.org/ip");
        string responseContent = await response.Content.ReadAsStringAsync();
        Console.WriteLine(responseContent);
      }
      catch (HttpRequestException e)
      {
        Console.WriteLine("Request failed with error: ", e.Message);
      }
    }
  }
}

The proxy server will respond with a 407: Proxy Authentication Required error when the credentials are invalid. The request will fail, and HttpClient will raise an HttpRequestException stating:

The remote server returned an error: (407) Proxy Authentication Required.

To avoid that, make sure the proxy username and password are correct.

Use a Rotating Proxy with HttpClient in C#

Even if you protect your IP with a proxy, the target server can still block your script if you make too many requests. The good news is you can avoid that with a rotating proxy approach.

That method involves using a new proxy after a specified period or number of requests, or randomly. Your end IP will keep changing, making the server unable to track you. That's the main advantage of an HttpClient proxy rotator!

Let's learn how to do so in C#.

Rotate IPs with a Free Solution

First, retrieve a pool of free proxies and store them in a list:

List<string> proxies = new List<string>
{
  "http://129.151.91.248:80",
  "http://18.169.189.181:80",
  // ...
  "http://212.76.110.242:80"
};

Then, define a function to execute a GET request through a rotating proxy. You'll extract a random proxy from the pool and use it to instantiate an HttpClient object, to then perform a GET request to the URL passed as a parameter.

static async Task<HttpResponseMessage> MakeRequestUsingRandomProxy(List<string> proxies, string url)
{
  // extract a random proxy from the list
  Random random = new Random();
  int index = random.Next(proxies.Count);
  string proxyURL = proxies[index];
  
  // set the proxy
  WebProxy webProxy = new WebProxy(proxyURL);
  HttpClientHandler handler = new HttpClientHandler()
  {
    Proxy = new WebProxy(proxyURL)
  };

  // make the request with the random proxy
  using (HttpClient client = new HttpClient(handler))
  {
    return await client.GetAsync(url);
  }
}

You can use MakeRequestUsingRandomProxy as below to randomize the proxy selection for each request. Every time you run the function below, you'll get a different IP.

HttpResponseMessage response = await MakeRequestUsingRandomProxy(proxies, "https://httpbin.org/ip");
string responseContent = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseContent);

Awesome! You built a proxy rotator, but it has a couple of issues:

1. It relies on free proxies: We used them to learn the basics, but you should never rely on them because they're failure-prone, slow, and get you blocked most of the time. Instead, take a look at our list of the best proxy types for scraping.
2. It instantiates several HttpClient times: Every time the function gets called, it creates a new HttpClient instance. The documentation advises against it. The reason is that HttpClient allocates a lot of sockets per instance, not disposing them for future reuse.

Thus, that script is unreliable and resource-intensive. You may think of caching an HttpClient instance and updating its proxy configurations. The library doesn't support that, so there isn't an easy fix for the resource leak.

To address the issues, you need to use commercial proxies and handle HttpClient instances with IHttpClientFactory, although it may end up being expensive and unnecessarily complex.

The real solution? A premium scraping proxy like ZenRows.

Rotating Premium Proxy for Scraping with HttpClient

Premium proxy providers used to be expensive, but that all changed with services like ZenRows. Starting at just $49 a month, you get access to reliable proxy servers with IP rotation and geo-location capabilities.

Sign up now for free to redeem your 1,000 API credits. After creating an account, reach the Request Builder page. Paste your target URL and check "Premium Proxy" to enable proxy rotation:

You'll get a URL you can use with HttpContent:

http://<YOUR_ZENROWS_KEY>:[email protected]:8001

Integrate it as below:

using System.Net;

namespace HttpClientProxy
{
  class Program
  {  
    static async Task Main(string[] args)
    {
      // ZenRows proxy URL
      string proxyURL = "http://<YOUR_ZENROWS_KEY>:[email protected]:8001";
      WebProxy webProxy = new WebProxy(proxyURL);
      HttpClientHandler handler = new HttpClientHandler()
      {
        Proxy = new WebProxy(proxyURL)
      };

      HttpClient client = new HttpClient();

      using HttpResponseMessage response = await client.GetAsync("https://httpbin.org/ip");
      string responseContent = await response.Content.ReadAsStringAsync();
      Console.WriteLine(responseContent);
    }
  }
}

Run the script several times, and you'll always see a different IP. The entire rotation logic has been boiled down to a single proxy call!

As a last remark, be aware that proxies are usually not enough to avoid getting blocked. That's why ZenRows gives you access to a whole toolkit for getting around anti-bot measures. It auto-rotates User-Agents and adopts many other techniques to bypass any anti-scraping system for you. Thus, also check the "Anti-bot" and "JavaScript Rendering" parameters for maximum effectiveness.

Well done! Now you have an effective and reliable C# HttpClient proxy rotator!

Conclusion

This step-by-step tutorial explained how to configure a proxy in HttpClient. You began with the basics and have become an HttpClient Proxy C# ninja!

Now you know:

• What an HttpClient proxy is.
• The basics of setting a proxy in C#.
• How to deal with an authenticated proxy with HttpClient.
• How to build a rotating proxy, why this solution doesn't work with free proxies and what the best practices are.

Remember that proxies help you avoid IP blocks, but advanced anti-scraping technologies can still detect you. The solution is ZenRows, a scraping tool with anti-bot bypass features and the best rotating residential proxies. Try it for free today!

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn. It looks like this tutorial has helped you. Be someone's hero today: Share on Twitter Share on LinkedIn