How to Set User Agent in Python Requests

The Requests library's default User Agent gives you away as a bot during scraping. But replacing it with a real browser's User Agent can reduce the risk of anti-bot detection.

In this tutorial, you'll learn how to change the User Agent in Python's Requests library and handle scaling with User Agent randomization.

What Is a User Agent in Python Requests?

The User Agent request header is a string describing the client making an HTTP request. It tells the server about the browser engine, type, version, and operating system.

The details in the User Agent are part of the browser fingerprint, which anti-bot technologies use to help determine whether a request is genuine or bot-like. This identification factor can significantly influence the outcome of a scraping request.

For example, a Google Chrome browser may have the following User Agent string:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36

The example User Agent above reveals that the requesting client is a version 137.0.0.0 Chrome browser running on a Windows 10 64-bit operating system with the WebKit rendering engine and compatible with the KHTML and Firefox Gecko engines.

Similarly, here's an example of a Firefox UA:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0

And you see a mobile UA next:

Mozilla/5.0 (iPhone; CPU iPhone OS 18_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Mobile/15E148 Safari/604.1

Your Python Request web scraper is denied access because its default User Agent outright reveals that it's a bot. Let's see how.

The following script sends an HTTP request to http://httpbin.io/user-agent, a web page that returns your User Agent string.

import requests

response = requests.get("http://httpbin.io/user-agent")

print(response.text)

The above code produces the request's default User Agent, as shown:

{
  "user-agent": "python-requests/2.31.0"
}

Look at the User Agent string python-requests/2.31.0, and you'll see that any website can tell it's from an automated script. That's why you have to specify a custom and well-formed User Agent for Python Requests.

Set User Agent in Python Requests

To set the Python Requests User Agent, you need to pass a dictionary containing the new string in your request configuration.

Start by importing the Python Requests library. Then create an HTTP headers dictionary containing the new User Agent, and send a request with your specified User-Agent string. The last step is printing the response.

# pip3 install requests
import requests
 
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"}
 
# request the target site with the User Agent
response = requests.get("http://httpbin.io/user-agent", headers=headers)
 
print(response.text)

If you get a result similar to the example below, you've succeeded in faking your User Agent.

{
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
}

Let's now see how to handle scaling with User Agent randomization.

Set Random User Agents With Python Requests

Using the same User Agent to access multiple pages on a domain makes it easier for a website’s anti-bot system to detect and block your scraper. To increase your scraper’s stealth, randomize the User Agent by selecting a different string from a list for each request.

You can rotate User Agents by shuffling and picking a random one per request using random.shuffle and itertools Python packages.

First, import random and itertools and create a User Agent list:

# ...
import random
from itertools import cycle

# create a User Agent list
user_agent_list = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
    "Mozilla/5.0 (iPhone; CPU iPhone OS 18_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Mobile/15E148 Safari/604.1",
    # ...,
]

Next, use random.shuffle to shuffle the User Agent list and create a generator from the randomized list using the itertools.cycle method. Get a random value from the pool with the next function. Then, use the generated User Agent in the request headers:

# ...

# shuffle the User Agent list
random.shuffle(user_agent_list)

# generate a single random User Agent from the pool
random_user_agent = cycle(user_agent_list)
user_agent = next(random_user_agent)

# use the randomized User Agent
headers = {"User-Agent": user_agent}

Update the previous code with these changes, and you'll get the following complete code:

# pip3 install requests
import requests
import random
from itertools import cycle

# create a User Agent list
user_agent_list = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
    "Mozilla/5.0 (iPhone; CPU iPhone OS 18_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Mobile/15E148 Safari/604.1",
    # ...,
]

# shuffle the User Agent list
random.shuffle(user_agent_list)

# generate a single random User Agent from the pool
random_user_agent = cycle(user_agent_list)
user_agent = next(random_user_agent)

# use the randomized User Agent
headers = {"User-Agent": user_agent}

# request the target site with the randomized User Agent
response = requests.get("http://httpbin.io/user-agent", headers=headers)

print(response.text)

Here's a sample result for four consecutive requests:

# request 1
{
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
}

# request 2
{
    "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 18_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Mobile/15E148 Safari/604.1"
}

# request 3
{
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0"
}

# request 4
{
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
}

Great! You've randomized User Agents using a custom technique.

However, this approach is only effective for a small number of requests, as anti-bot systems can still detect your scraper through other bot-like signals.

Let's see the proper approach to randomize User Agents and bypass anti-bot measures at scale during web scraping.

How to Rotate Infinite User Agents at Scale

Creating a reliable User Agent rotation system is more complex than it seems. Beyond maintaining thousands of strings, you'll need to constantly update browser versions, validate OS compatibility, and remove suspicious combinations.

Also, relying only on User Agents is insufficient to bypass anti-bot systems. Modern websites look beyond the User Agent to identify automation by checking your IP reputation, request patterns, header consistency, TLS fingerprinting, etc.

The most effective solution is a web scraping solution, such as the ZenRows Universal Scraper API. With a single API call, ZenRows auto-rotates up-to-date User Agents and handles anti-bot bypass behind the scenes. This lets you focus on your core business logic rather than wasting time and resources battling unprecedented scraper failures.

Let's see how ZenRows performs against a protected page like the Antibot Challenge page.

Start by signing up for a new account, and you'll get to the Request Builder. Paste the target URL in the link field, enable JS Rendering, and activate Premium Proxies.

Next, select Python and click on the API connection mode. Then, copy the generated code and paste it into your script.

# pip3 install requests
import requests

url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
    "url": url,
    "apikey": apikey,
    "js_render": "true",
    "premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params, print(response.text)

Run the code, and you'll successfully access the page:

<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! 🎉 You’ve successfully bypassed the anti-bot challenge page using ZenRows. This works for any website.

Conclusion

In this guide, you've learned the essentials of managing User Agents at scale with Python Requests. You now understand the following:

• What makes a well-formed User Agent string and why it matters.
• Setting custom User Agents in your Python requests.
• Implementing basic User Agent rotation with Python's random module.
• Why User Agent management alone isn't enough for reliable web scraping.

While sustainable User Agent management is essential during web scraping, remember that anti-bots look beyond the User Agent to detect bots. So you need more than setting a custom User Agent to handle today's modern websites. We recommend integrating ZenRows for an auto-managed solution that bypasses any anti-bot measures at scale. ZenRows ensures you extract all the data you need without getting blocked.

Try ZenRows for free!