If you’ve ever tried scraping at scale, you know anti-bot measures are getting smarter and meaner. At Clay, they hit a wall. Their enrichment automations started failing, and debugging their scraping layer became a full-time job.
In this post, we’ll walk you through how Clay scaled through initial blocks, what failed, and why ZenRows became its go-to for bypassing bot defenses. You’ll see their process, the metrics used, and practical tips for integrating ZenRows into your own stack.
The Problem: Anti-Bot Measures Blocked Users
Many developers and sales teams rely on Clay's data enrichment automation for lead generation, competitive monitoring, price comparison, product research, and other purposes.
Like many automation platforms, Clay abstracts away the scraping layer so users can focus on results and not bother about the technical hurdles of getting data. However, important targets began to implement advanced anti-bot measures, blocking Clay's native scraping solutions.
This meant that the native scrapers returned zero data when targeting these heavily protected sites.
When scraping a large volume of data, an error log like the following isn't a pleasant sight for any developer. It diverts attention from building to debugging.
{
"status": "error",
"error_code": "SCRAPER_BLOCKED",
"message": "Target site refused connection due to bot detection. Please review anti-bot mitigation strategies.",
"details": {
"http_status": 403,
"target_url": "https://target-website.com/data",
},
}
This was the ordeal of Clay and its power users who were feeding their pipelines with data from heavily protected websites. Customer tickets about this issue peaked, diverting the engineering team’s attention away from core development work. This problem highlighted the urgent need for a sustainable, auto-managed solution to scrape these hard targets effectively.
Why Native Scrapers Get Blocked
Clay's native web scrapers were failing, not because their scraping logic was faulty, but due to external protective measures deployed by data sources. In fact, their data extraction logic was state-of-the-art, written by experienced developers.
The issues identified as the source of the anti-bot blocks include the following:
- Native scrapers struggled to keep up with frequent anti-bot security updates.
- Websites that were previously easy to scrape started implementing anti-bot measures, which quickly disrupted running data pipelines.
- Internal scrapers failed JavaScript challenges and advanced CAPTCHA tests.
- Fingerprinting measures deployed by web application firewalls (WAFs) profiled native scrapers, further preventing access to anti-bot-protected sites.
- Native scrapers failed sophisticated human behavioral checks, such as background analysis powered by machine learning.
Clay's internal engineering team had the intellectual capacity to handle these never-ending loopholes. Outsourcing hard-target scraping to experts, however, proved more impactful than wasting time and resources on continuous patching and debugging of unpredictable failures.
Let's see how they approached these issues.
Clay's Hands-On Approach to Scrape Without Getting Blocked
Besides evading anti-bots, scalability, speed, and reliability were also crucial for success.
Clay's strategy to bypass anti-bot measures and address performance bottlenecks was straightforward. After days of benchmarking different services, they turned to ZenRows, the one web scraping solution that worked best for their use cases and expectations.
Since the ZenRows Universal Scraper API requires only a simple API call, Clay took two unique steps:
- They leveraged the Universal Scraper API internally, enabling access to tougher targets for their most demanding use cases.
- They further collaborated directly with ZenRows to launch a ZenRows Scrape action on their platform. This enables users to define custom parameters and data points using their personal ZenRows credentials.
"One of the things I love about ZenRows is that you don't need to configure it on your own. You can simply use it through Clay." — Yash Tekriwal, Founding Educator at Clay.
Integrating ZenRows into your existing scraping script is easy and involves the following steps:
- Sign up and go to your ZenRows Request Builder.
- Paste your target URL in the link box, and activate Premium Proxies and JS Rendering to effectively bypass anti-bots.
- Choose your preferred programming language and select the API connection mode.
- Copy and paste the generated code into your scraper script.
Here's a sample generated code that scrapes the Anti-bot Challenge page, a heavily protected site:
# pip3 install requests
import requests
url = "https://www.scrapingcourse.com/antibot-challenge/"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
If you run the above code with your ZenRows API key, you'll get the following response, showing you bypassed the site's security check:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
You’ve now seen how easy it is to integrate ZenRows and scrape any website without getting blocked.
With such a simple setup, Clay could now fully channel engineering time and resources into core business priorities, such as innovating and scaling its automation platform. Users who were previously blocked now scrape even the most protected websites reliably at scale without limitations.
"ZenRows has emerged as a go-to solution for many of our users, including myself, simply because it's so powerful. What sets ZenRows apart is its exceptional ability to bypass advanced anti-scraping measures." — Clay's Founding Educator.
Results: From Downtime to Millions of Data Points in Real-Time
ZenRows not only enabled Clay to bypass anti-bot measures but also significantly increased scraping performance. For Clay power users like Eric Nowoslawski, ZenRows stands out as the top lead generation tool for obtaining high-quality leads from protected sources.
We power it all with ZenRows. It's the most cost-effective and reliable data enrichment tool on the market.
After integrating ZenRows, Clay's enrichment performance drastically improved from struggling with failed scraping requests to generating millions of data points from several targets in real-time.
With a scraping pipeline that now reliably pulls high-quality information, data preparation time has decreased from weeks to minutes. Scrapers that once required frequent manual intervention were now fully automated, allowing the team to focus on higher-level tasks and decision-making.
Takeaways and Advice for Developers
Here are the key lessons you should take away from Clay’s approach to overcoming anti-scraping measures:
- Unprotected sites could suddenly deploy anti-scraping measures, causing your existing scraping script to fail abruptly.
- The cost of building and maintaining in-house web scrapers can be high, especially when you need to bypass advanced anti-bot measures or scale up to millions of data points in real-time.
- Even if you develop an in-house scraping solution, websites' anti-bot updates can quickly render your approach ineffective. This results in blocked requests, incomplete data, and a recurring cycle of maintenance.
- Outsource overly technical tasks, such as anti-bot bypass, to a reliable scraping solution rather than diverting development time and resources to debugging unprecedented failures.
- Opt for a scraping solution with proven reliability and a high success rate. ZenRows, for example, has a high success rate of up to 99.93%, making it a strong choice for developers who need dependable data access.
- Your chosen web scraping solution should be easily integrable into your existing system or code.
Conclusion
You've learned how Clay dramatically transitioned from getting blocked to scaling their scraping operations in real-time without limitations. Their approach was simple, yet highly impactful and timely.
With ZenRows, you get an auto-managed and auto-scaled scraping infrastructure that proactively bypasses even the most advanced anti-bot measures before they can disrupt your data pipeline. If you’re looking to future-proof your own web scraping projects and keep them running smoothly without interruptions, consider exploring a solution like ZenRows.