Trying to avoid bot detection and scrape data without constant blocks? You're not alone. With anti-bot systems getting smarter, scraping has become a lot trickier - but there are ways to stay ahead.
Quick Solution: Web Scraping Without Getting Blocked
The easiest and most reliable way to avoid anti-bot detection sustainably is to use a web scraping solution like the ZenRows Universal Scraper API. It comes with all the toolkits you need to avoid getting blocked.
Let's see how ZenRows works with a heavily protected site like the Antibot Challenge page.
To start, sign up for free to open the Universal Scraper API Request Builder. Paste your target URL in the address box. Then, activate Premium Proxies and JS Rendering.
Choose your programming language (Python, in this case) and choose the API connection mode. Copy the generated code and paste it into your script.
The generated Python code looks like this:
# pip3 install requests
import requests
url = "https://www.scrapingcourse.com/antibot-challenge"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
The above code accesses the protected site and extracts its full-page HTML, as shown:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! You can now confidently bypass any anti-bot measure with only a few code lines.
4 Additional Strategies to Bypass Bot Detection
Let's explore 4 more strategies.
Use Proxies
Many websites use IP tracking to monitor the traffic from an IP address. This anti-bot measure allows them to detect and ban IPs that violate specific access rules, such as rate limiting and geo-restrictions.
Web scrapers are prone to such bans because they usually send multiple requests simultaneously, even to localized content unavailable within the client's region.Â
Proxies are an excellent solution to mitigate this, as they route your requests through another IP address. This way, the target website sees you as another user.
While most scraping libraries across different programming languages support proxies, the code below shows a simple way to set it up with Python's Requests:
# pip3 install requests
import requests
# specify the proxy address
proxies = {
"http": "http://200.174.198.86:8888",
"https": "http://200.174.198.86:8888",
}
# request the target website with proxy
response = requests.get("https://httpbin.io/ip", proxies=proxies)
# print the response
print(response.text)
Of all the proxy solutions available, residential proxies are the best for large-scale web scraping. They offer residential IP addresses assigned to network users by internet service providers (ISPs).
You should also opt for a residential proxy provider that offers proxy rotation out of the box. This gives you additional reliability by rotating real user IPs from a pool, allowing each scraping request to appear as a natural user. Most residential proxy services also feature geolocation, which enables you to switch regions and access geo-restricted content.
Mimic User Behavior
Anti-bot measures often deploy algorithms to monitor user behavior, including clicks, mouse movements, scrolling patterns, etc.Â
However, web scraping typically has a predictable flow, even if you use automation tools like Selenium. These include clicking the same element, visiting several pages simultaneously, rapidly filling out forms, and scrolling the same height. Such fixed actions deviate from natural usage patterns and can trigger anti-bot measures.Â
One way to avoid detection during scraping is to randomize web interactions to mimic human behavior. These include varying the scroll heights, clicking random elements, using adequate retry mechanisms such as exponential backoffs, and more.
The following snippet is a basic scraper demonstrating how to simulate clicking with Selenium. It visits the E-commerce Challenge website and prints the current page title before and after clicking the first product's name.
# pip3 install selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
# initialize the Chrome browser in headless mode
chrome_options = Options()
chrome_options.add_argument("--headless=new")
# set up the WebDriver
driver = webdriver.Chrome(options=chrome_options)
# open the target website
driver.get("https://www.scrapingcourse.com/ecommerce/")
# get the page title before clicking an element
print(f"Page title before click: {driver.title}")
# obtain the first product element's name
first_product_name = driver.find_element(By.CLASS_NAME, "product-name")
# click the first product to simulate human interaction
first_product_name.click()
# get the title of the current page
print(f"Page title after click: {driver.title}")
# close the browser
driver.quit()
The above is a starting point using Selenium. You can also simulate user interactions using other headless browsers like Puppeteer and Playwright.
Keep in mind that while this technique can help you bypass basic checks, it's insufficient against advanced anti-bot measures. That brings you to the next solution.
Implement Fortified Headless Browsers
Conventional headless browsers like Selenium and Puppeteer are prone to anti-bot detection if you don't modify their default configurations. They present bot-like attributes, such as a WebDriver automation flag in the navigator field, a HeadlessChrome User Agent flag in headless mode, and the absence of specific browser attributes used for fingerprinting.
While you can manually patch these missing pieces, the process can be time-consuming and technical. Fortunately, these headless browsers have third-party plugins or helpers that remove the bot flags and reduce the chances of detection.Â
For instance, SeleniumBase with Undetected ChromeDriver is a good solution that can help you avoid anti-bot detection in Selenium. You can also use the Puppeteer Stealth plugin for Puppeteer.
Let's see how SeleniumBase with Undetected ChromeDriver works with the Antibot Challenge page. The script below opens the protected website in non-headless mode and clicks the CAPTCHA challenge if present.
# pip3 install seleniumbase
from seleniumbase import Driver
# initialize driver in GUI mode with UC enabled
driver = Driver(uc=True, headless=False)
# set the target URL
url = "https://www.scrapingcourse.com/antibot-challenge"
# open URL using UC mode with 6 second reconnect time
driver.uc_open_with_reconnect(url, reconnect_time=6)
# attempt to bypass CAPTCHA if present using UC mode's built-in method
driver.uc_gui_click_captcha()
# take a screenshot of the current page and save it
driver.save_screenshot("screenshot.png")
# close the browser and end the session
driver.quit()
The above code accesses the protected page and clicks the CAPTCHA challenge to bypass the anti-bot measure.
Want to learn more? Check out our detailed article on avoiding anti-bot detection in Selenium.
However, a significant shortcoming of this solution is that these fortified headless browser tools keep losing their evasion capabilities to anti-bot measures. Due to their open-source nature, anti-bots constantly monitor and block their evasion strategies.
Set Custom User Agent
The User Agent is an essential web scraping header. It plays a vital role in browser fingerprinting, describing the client's information, including the browser version, name, supported platform, and rendering engine.
Websites typically assess an incoming request's User-Agent header for signs of automation, such as inconsistencies in browser versioning, default HTTP client strings, and other anomalies. Unfortunately, most scraping tools leave bot-like traces in their default User Agent header, making them vulnerable to detection.Â
For example, Python's Requests library has the following bot-like User Agent:
{
"user-agent": "python-requests/2.31.0"
}
Such User Agent strings raise suspicion and can result in blocking. One way to avoid getting blocked is to set a custom User Agent from a real browser like Chrome.
The following code demonstrates how to set a Chrome User Agent using Python's Requests:
# pip3 install requests
import requests
# define the headers
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
}
# request the target website
response = requests.get("https://httpbin.io/user-agent", headers=headers)
# print the response
print(response.text)
Beyond setting a single User Agent, rotating it to simulate different devices can improve your chances of bypassing anti-bot detection during scraping. However, when rotating the User Agent header, ensure consistency with related client-hint headers like Sec-Ch-Ua and Sec-Ch-Ua-Platform, as mismatches can raise suspicion.
Conclusion
You've learned 5 solid techniques for bypassing bot detection when scraping. While solutions like proxies, mimicking user behavior, and headless browser fortification aren't self-sufficient against anti-bot measures, combining them can yield better results.
That said, the best way to avoid getting blocked is to use a web scraping solution like ZenRows. It offers an all-in-one toolkit to avoid blocks at any scale, including real-user behavior simulation, automatic premium proxy rotation, anti-bot auto-bypass, and more. Try ZenRows for free now!
Frequent Questions
1. Why Am I Being Detected as a Bot?
You may be detected as a bot due to:
- Repeated requests from the same IP address.
- Bot-like User Agent headers or missing browser attributes.
- Predictable interaction patterns, such as rapid clicks or scrolling.
- Using unpatched headless browsers with automation flags.
2. How Can I Disable Bot Detection?
You cannot disable bot detection. However, you can reduce the chances of being detected by:
- Using rotating proxies to avoid IP bans.
- Mimicing human behavior by randomizing interactions like clicks and scrolls.
- Using fortified headless browsers like SeleniumBase or Puppeteer Stealth.
- Setting custom and consistent User Agents to avoid detection.
3. What Are the Best Tools to Avoid Bot Detection?
The best tools include:
- ZenRows Universal Scraper API: Offers premium rotating proxies and top-notch anti-bot bypass.
- SeleniumBase with Undetected ChromeDriver: Fortifies Selenium for better evasion.
- Puppeteer Stealth Plugin: Reduces bot-like signals in Puppeteer.
4. How Do Proxies Help Bypass Bot Detection?
Proxies mask your real IP by routing requests through different addresses. Using rotating residential proxies ensures each request appears to come from a unique, legitimate user, reducing the chances of detection and bans.
5. How Can I Mimic Human Behavior While Scraping?
To mimic human behavior:
- Randomize interaction patterns like clicks, scrolls, and delays.
- Use headless browsers to simulate real user actions.
- Add pauses and retries to mimic natural browsing behavior.