How to Set Up User Agent With Net/http in Golang

June 17, 2024 ยท 8 min read

Are you looking to specify a custom Golang net/http user agent? You're in the right place. With websites employing various techniques to regulate bot traffic, you must mimic natural browser behavior to access your desired data.

In this article, you'll learn how to configure your net/http user agent to emulate an actual browser.

What Is the Net/http User Agent?

The net/http User Agent (UA) string is critical to the HTTP headers sent with every HTTP request. These headers are metadata that convey additional information to the web server, such as authentication credentials, content type, caching, etc.

Most importantly, the user agent provides details about various components of the web client, such as browser name, version, device type, and operating system.

Here's a sample Google Chrome browser UA string:

Example
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36

It indicates that the user uses Chrome version 109.0.0.0, which employs the WebKit rendering engine on a 64-bit Windows 10 (Windows NT 10.0) platform.

However, your Golang net/http user agent typically looks like this.

Example
Go-http-client/1.1

You can see yours by making a basic request to httpbin.io/user-agent.

The clear distinction of the two UA samples above confirms how easy it is for websites to differentiate between a net/http request and an actual browser. That's why specifying a custom user agent is essential.

Let's see how.

How to Set Up a Custom User Agent in Net/http

To specify a User-Agent header in your Go application using net/http, follow the steps below:

1. Prerequisites

The net/http package is part of Go's standard library, included with every Go installation by default. So, there's no need for a separate installation. You only need to create your Go project and import the necessary dependencies.

Once you have everything set up, you'll be ready to write your code. Here's a basic script that makes a GET request to https://httpbin.io/user-agent and retrieves its text content.

program.go
package main
 
// import the necessary dependencies
import (
    "fmt"
    "io"
    "net/http"
)
 
func main() {
    // make an HTTP GET request
    response, err := http.Get("https://httpbin.io/user-agent")
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer response.Body.Close()
 
    // read the response body
    body, err := io.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // print the text content
    fmt.Println(string(body))
}

2. Customize the UA

net/http provides a Header.Set() method that allows you to set HTTP headers. This method takes two parameters: the header's name and its value. In this case, you would pass "User-Agent" as the header name and your desired UA string as the header value.

To do that, you must create a Client. This gives you more control over HTTP headers and allows you to manipulate them according to your needs.

Let's break it down into smaller steps using the Google Chrome UA sample provided earlier and httpbin as the target website.

Import the necessary dependencies and create a custom HTTP Client

program.go
package main
 
import (
    "fmt"
    "io"
    "net/http"
)
 
func main() {
    // create custom HTTP client
    client := &http.Client{
        Transport: &http.Transport{},
    }
}

After that, create your GET request and set the custom user agent using Header.Set().

program.go
func main() {
    //...
 
    // create HTTP request
    req, err := http.NewRequest("GET", "https://httpbin.io/user-agent", nil)
    if err != nil {
        // Handle error
        return
    }
 
    // set User-Agent header
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36")
}

Next, make the HTTP request, close the response body, read it, and print its text content to the console.

program.go
func main() {
    //...
    
    // make HTTP request
    response, err := client.Do(req)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    // close the response body
    defer response.Body.Close()
 
    // read the response body
    body, err := io.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // print the text content
    fmt.Println(string(body))
}

Putting it all together, here's the complete code:

program.go
package main
 
import (
    "fmt"
    "io"
    "net/http"
)
 
func main() {
    // create custom HTTP client
    client := &http.Client{
        Transport: &http.Transport{},
    }
 
    // create HTTP request
    req, err := http.NewRequest("GET", "https://httpbin.io/user-agent", nil)
    if err != nil {
        // Handle error
        return
    }
 
    // set User-Agent header
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36")
 
    // make HTTP request
    response, err := client.Do(req)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    // close the response body
    defer response.Body.Close()
 
    // read the response body
    body, err := io.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // print the text content
    fmt.Println(string(body))
}

Run it, and your response should be the predefined custom User Agent.

Output
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
}

Congrats! You've set your first Golang net/http user agent.

Not to spoil the party, but specifying a single UA may not be enough in most cases, as websites can eventually identify your scraper. However, you can fix this issue by rotating between user agents.

3. Use a Random User Agent in Net/http

Websites interpret multiple requests from the same user agent as automated traffic and block them accordingly. But by rotating UAs, you can appear to the web server as though your requests come from different browsers (users).

When making HTTP requests using the net/http package, you can rotate between UAs by randomly selecting from a predefined list.

Here's how you can modify your previous code to achieve this.

Start by importing the necessary dependencies. Then, define a list of User-Agent strings. You can include various UAs to emulate different browsers, operating systems, and devices. We've selected a few from this list of web-scraping user agents for this example.

program.go
package main
 
// import the necessary dependencies
import (
    "fmt"
    "io"
    "net/http"
    "math/rand"
)
 
func main() {
    // list of User-Agent strings
    var userAgents = []string{
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
    // add more UA strings as needed
    }
}

Next, select a random UA from the list. To do this, generate a random integer and use it to choose one from the list.

program.go
func main() {
    //...
 
    // select a random UA from the list
    randomIndex := rand.Intn(len(userAgents))
    randomUA := userAgents[randomIndex]
}

After that, create a Client and HTTP request. Then, specify the randomly selected UA using Header.Set().

scraper.go
func main() {   
    //...
 
    // create custom HTTP client with custom Transport
    client := &http.Client{
        Transport: &http.Transport{},
    }
 
    // create HTTP request
    req, err := http.NewRequest("GET", "https://httpbin.io/user-agent", nil)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // set User-Agent header
    req.Header.Set("User-Agent", randomUA)
 
}

Lastly, update the previous code with the steps above to get the following complete code.

scraper.go
package main
 
// import the necessary dependencies
import (
    "fmt"
    "io"
    "net/http"
    "math/rand"
)
 
func main() {
    // list of User-Agent strings
    var userAgents = []string{
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
    // add more UA strings as needed
    }
 
    // select a random UA from the list
    randomIndex := rand.Intn(len(userAgents))
    randomUA := userAgents[randomIndex]
 
    // create custom HTTP client with custom Transport
    client := &http.Client{
        Transport: &http.Transport{},
    }
 
    // create HTTP request
    req, err := http.NewRequest("GET", "https://httpbin.io/user-agent", nil)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // set User-Agent header
    req.Header.Set("User-Agent", randomUA)
 
    // make HTTP request
    response, err := client.Do(req)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    // close the response body
    defer response.Body.Close()
 
    // read the response body
    body, err := io.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // print the text content
    fmt.Println(string(body))
}

Every time you run the script, a different UA will be used to make your request. For example, here are the results for three requests:

Output
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
}
 
{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"
}
 
{
  "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
}

Awesome! You've successfully rotated your Golang net/http user agent.

It's worth noting that real-world use cases would require a larger UA list. So, paying attention to your UA construction is essential. Well-formed UAs are critical to avoid triggering anti-bot measures.

For example, your UA string must match other HTTP headers.

If the User-Agent string identifies the client as a particular browser and version, but other HTTP headers suggest different characteristics or behaviors, it might raise suspicion.

That said, constructing and maintaining a list of well-formed UAs can take time and effort. But no worries, the following section provides a more straightforward solution.

Avoid Getting Blocked When Scraping With Net/http

While specifying browser-like user agents is crucial in avoiding detection, it may not be sufficient. Websites employ various measures beyond checking the UA to detect and block scraping activities.

A more comprehensive approach is often required to overcome these challenges and avoid getting blocked while scraping. One popular quick fix is using web scraping residential proxies.

These proxies route your scraping requests through real residential IP addresses, making them appear to originate from genuine users rather than automated bots. It helps bypass CAPTCHAs and other detection mechanisms implemented by websites.

However, dealing with anti-bot systems can get complex, and more than residential proxies may be needed. See for yourself. Try to scrape a protected web page (https://www.g2.com/products/visual-studio/reviews) using the previous custom user agent script.

Visual Studio Homepage
Click to open the image in full screen

You'll get the following result.

program.go
package main
 
import (
    "fmt"
    "io"
    "net/http"
)
 
func main() {
    // create custom HTTP client
    client := &http.Client{
        Transport: &http.Transport{},
    }
 
    // create HTTP request
    req, err := http.NewRequest("GET", "https://www.g2.com/products/visual-studio/reviews", nil)
    if err != nil {
        // Handle error
        return
    }
 
    // set User-Agent header
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36")
 
    // make HTTP request
    response, err := client.Do(req)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    // close the response body
    defer response.Body.Close()
 
    // read the response body
    body, err := io.ReadAll(response.Body)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
 
    // print the text content
    fmt.Println(string(body))
}

This result confirms that specifying a Golang net/http user agent isn't enough. The script was met with Cloudflare anti-bot protection that denied you access.

In this case, using specialized tools can simplify the process. One such solution is ZenRows, a web scraping API that offers everything you need to scrape without getting blocked.

ZenRows automatically rotates User-Agents, manages residential proxies, and employs valuable techniques to handle challenges posed by anti-bot systems. With ZenRows, you can focus on extracting data without worrying about getting blocked while web scraping in Golang.

Here's how ZenRows works against a heavily protected website (https://www.g2.com/products/visual-studio/reviews`).

First, sign up for your free API key, and you'll be directed to the Request Builder page.

Paste your target URL (https://www.g2.com/products/visual-studio/reviews), check the box for Premium Proxies, and activate the JavaScript Rendering boost mode. Then, Select Go as the language you'll use to get your request code generated on the right.

ZenRows Request Builder
Click to open the image in full screen

ZenRows integrates with net/http and provides the boilerplate code. Copy this code to your preferred editor. Your code should look like this:

Output
<!DOCTYPE html>
 
<body>
    <div class="cf-wrapper cf-header cf-error-overview">
        <h1 data-translate="block_headline">Sorry, you have been blocked</h1>
        <h2 class="cf-subheadline"><span data-translate="unable_to_access">You are unable to access</span> g2.com</h2>
    </div>
    <!-- 
    ...
    -->
 
</body>

Run it, and you'll get the target page's HTML.

Output
<!DOCTYPE html>
<head>
    <title>Visual Studio Reviews 2024: Details, Pricing, &amp; Features | G2</title>
    <!--
    ...
    -->
</head>z

Cool, right? Bypassing anti-bot systems is easy with ZenRows.

Conclusion

Specifying a custom Golang net/http user agent can help avoid detection. However, relying solely on UAs, may not be sufficient, as websites employ various other measures to detect and block scraping activities. Not to mention how challenging crafting and maintaining accurate User Agent strings can be.

To streamline the scraping process and avoid the hassle of finding and configuring UAs, consider ZenRows.

Ready to get started?

Up to 1,000 URLs for free are waiting for you