Using an incorrect User Agent when scraping or not applying some related best practices is a recipe for getting blocked. To solve that, you'll find here a list of the best User Agents for scraping and some tips on how to use them.
What Is a User Agent?
A User Agent (UA) is a string sent by a user's web browser to a server. It's located in the HTTP header and identifies the browser type and version, as well as the operating system. Accessed via JavaScript on the client side via the navigator.userAgent property, this information is used by the remote web server to identify and render content in a way compatible with the user's specifications.
While the User Agent contains different information, most web browsers follow the same structural format:
Mozilla/5.0 (<system-information>) <platform> (<platform-details>) <extensions>
For example, a User Agent string for Chrome (Chromium) might be the following:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36
Let's break it down: it contains the browser name (Chrome), the version number (144.0.0.0), and the operating system the browser is running on (Windows NT 10.0, 64-bit processor).
Why Is a User Agent Important for Web Scraping?
Since UA strings help web servers identify the type of browser (and bots) sending requests, adopting a real browser's string for scraping can help mask your spider as a web browser.
So you can bypass protection systems like Cloudflare with the right User Agent strings.
But using a misconfigured User Agent will almost always block your data extraction script.
What Are the Best User Agents for Scraping?
We compiled a list of User Agents that are effective for scraping. They can help you emulate a browser and avoid getting blocked:
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0
- Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 OPR/126.0.0.0 (Edition Campaign 34)
- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36
- Mozilla/5.0 (Macintosh; Intel Mac OS X 14.4; rv:145.0) Gecko/20100101 Firefox/145.0
- Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.1 Safari/605.1.15
- Mozilla/5.0 (Macintosh; Intel Mac OS X 14_4_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 OPR/126.0.0.0
- Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36
- Mozilla/5.0 (X11; Linux i686; rv:124.0) Gecko/20100101 Firefox/145.0
- Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0
How to Check User Agents and Understand Them
The easiest way to check and understand your User Agent is to visit UserAgentString.com. It automatically displays the User Agent for your web browsing environment. You can also get comprehensive information on other User Agents. You just have to copy/paste any string in the input field and click on ''Analyze.''
How to Set a New User Agent Header in Python?
Let's run a quick example by changing a scraper's User Agent using Python requests. We'll use a string associated with Chrome:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36
Use the following code snippet to set the User-Agent header while sending the request:
import requests
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36"}
# You can test if your web scraper is sending the correct header by sending a request to HTTPBin
response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.text)
Its output will look like this:
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36",
"X-Amzn-Trace-Id": "Root=1-63c42540-1a63b1f8420b952f1f0219f1"
}
}
And that's it! You now have a new User Agent for scraping.
How To Rotate Infinite User Agents at Scale
Maintaining a reliable User Agent system is more complex than just having a list of strings. You need a continuous process to update browser versions, verify operating system compatibility, and remove problematic combinations.
Beyond checking your User Agents, modern websites employ sophisticated detection methods to spot automation. They analyze your network behavior, verify header consistency across requests, check IP reputation scores, examine browser fingerprints, and more. Even with perfect User Agent management, these other detection methods can still block your scraper.
The most effective solution is to use a web scraping solution, such as the ZenRow Universal Scraper API. It provides auto-rotating up-to-date User Agents, premium proxy, JavaScript rendering, CAPTCHA auto-bypass, and everything you need to avoid getting blocked, regardless of the scale.
Let's see how ZenRows performs on a protected page, such as the Antibot Challenge page.
Start by signing up for a new account. Then open the Request Builder. Paste the target URL into the link field, then enable Adaptive Stealth Mode.
Next, select Python and click on the API connection mode. Copy the generated code and paste it into your script.
# pip3 install requests
import requests
url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"mode": "auto",
}
response = requests.get("https://api.zenrows.com/v1/", params=params, print(response.text)
The generated code uses Python's Requests library as the HTTP client. You can install this library using pip:
pip3 install requests
Run the code, and you'll successfully access the page:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! 🎉 You’ve successfully bypassed the anti-bot challenge page using ZenRows. This works for any website.
Conclusion
In this guide, you've learned the essentials of managing User Agents with Python Requests:
- Understanding the User Agent string structure.
- Selecting appropriate User Agents for different browsers.
- Best practices for User Agent implementation.
- Why User Agent management alone isn't enough for reliable web scraping.
Keep in mind that many websites use different anti-bot mechanisms to prevent web scraping. The most reliable way to bypass scraping blocks at scale is to use a web scraping solution, such as ZenRows. Integrate ZenRows today and extract all the data you need without getting blocked.
Try ZenRows for free now or speak with sales!
Frequent Questions
How can I change my User Agent when scraping?
To automatically change your User Agent for scraping, create an updated list of valid User Agents from real browsers on various operating systems. Then implement logic to rotate each User Agent string for a specified number of requests.
That said, ensure your User Agent string matches other request header strings, such as the Sec-Ch-Ua and Sec-Ch-Ua-Platform. Rotating these headers together is ideal than just focusing on the User Agent string. A mismatch between these header strings can resilt in blocking.
Does changing the User Agent prevent anti-bot detection?
Just rotating or changing the User Agent alone doesn't fully protect you from being blocked. Beyond changing the User Agent or tweaking scraping request headers, Anti-bot bypass requires continuous infrastructure maintenance and evasion tweaks to maintain stealth. To handle anti-bot evasion automatically and save time and resources, it's best to use a web scraping API.
What is a User Agent in web scraping?
A User Agent is a request header string that describes the client (a browser like Chrome or an HTTP client like Requests) sending a request to a server. It details the client's browser type, version, and operating system.
Some websites block certain User Agents, such as those from common HTTP clients like Requests or Axios, resulting in scraping blocks. It's essential to replace such User Agents with valid, real browser strings to reduce the likelihood of detection as a bot during scraping.
What is the most common User Agent?
The most common User Agent usually reflects the current version of a browser. Since Chrome is widely used by regular users, its current version on various platforms, including Windows, Linux, Android, iOS, and macOS, is considered common. This doesn't mean other User Agents are less useful for scraping. Up-to-date User Agent strings from browsers such as Firefox, Edge, and Safari are also highly valuable for web scraping.