Crawlee, by itself, is a great web scraping library. But how do you make sure that you’re getting the most out of its capabilities?
If your requests are considered bot-like or excessive, websites can block your IP address. But with a Crawlee proxy, you can prevent IP bans and access geo-restricted content.
Read on to find out how to set up a proxy in Crawlee. You will also learn a few best practices and tips to pick the right proxy provider to help you avoid getting blocked.
How to Set Up a Proxy in Crawlee
Crawlee offers native proxy support that makes it easy to manage proxies and route requests through proxy servers.
Here's a step-by-step guide on how to configure a Crawlee proxy:
You'll need proxies to follow along in this tutorial; you can grab one from the Free Proxy List website.
Also, we'll use Crawlee's CheerioCrawler class and the HTTPBin IP endpoint to verify the proxy configuration.
Let's begin!
Crawlee provides the ProxyConfiguration class for configuring connections to proxy servers using the constructor function based on specified options.
To do this, instantiate a new ProxyConfiguration object using the constructor and provide your proxy credentials through the proxyUrls option.
Below is a basic crawler demonstrating the steps above.
It creates a new instance of the ProxyConfiguration class by passing a single proxy URL. It then uses this instance in a Cheerio crawler that makes a request to a target website and logs its JSON response.
// import the required classes
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
// create a new ProxyConfiguration instance and specify your proxy URL
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: ['http://138.68.235.51:80'],
});
const startUrls = ['https://httpbin.org/ip'];
// initialize a CheerioCrawler class
const crawler = new CheerioCrawler({
// pass in proxy configuration instance
proxyConfiguration,
// modify requestHandler to log page content
async requestHandler({ request, body }) {
console.log(`Scraping: ${request.url}`);
const jsonData = JSON.parse(body);
console.log(jsonData);
},
});
await crawler.run(startUrls);
Here's the result:
INFO CheerioCrawler: Starting the crawler.
Scraping: https://httpbin.org/ip
{ origin: '138.68.235.51' }
Awesome! Right?
Note that the proxies used in this tutorial are free, often short-lived, and may not work at the time of reading this. You should get some new ones from the Free Proxy List website.
Alternatively, if you're using paid proxies, you'll likely need an authentication; more on this in the next section.
Proxy Authentication in Crawlee
Paid proxy providers often require additional credentials, such as a username and password, before giving users access to their servers.
In Crawlee, you can authenticate your proxy by adding the required credentials along with your proxy URL in the format below:
<PROXY_PROTOCOL>://<YOUR_USERNAME>:<YOUR_PASSWORD>@<PROXY_IP_ADDRESS>:<PROXY_PORT>
If the proxy in the previous example requires authentication, your code would look like this:
// import the required classes
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
// create a new ProxyConfiguration instance and specify your proxy URL
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: ['http://<YOUR_USERNAME>:<YOUR_PASSWORD>@138.68.235.51:80'],
});
const startUrls = ['https://httpbin.org/ip'];
// initialize a CheerioCrawler class
const crawler = new CheerioCrawler({
// pass in proxy configuration instance
proxyConfiguration,
// modify requestHandler to log page content
async requestHandler({ request, body }) {
console.log(`Scraping: ${request.url}`);
const jsonData = JSON.parse(body);
console.log(jsonData);
},
});
await crawler.run(startUrls);
Rotating Proxies in Crawlee
A single proxy can help you hide your original IP address, but websites often employ rate-limiting technologies that can block your proxy once you cross a certain threshold of requests.
As a result, you must rotate between proxies. This allows you to spread traffic over multiple servers so that each request seems like a unique user.
Crawlee's ProxyConfiguration class covers all the proxy-related use cases. You only need to pass a list of proxy URLs to the proxyUrls option, and ProxyConfiguration handles rotating through them in a "round-robin" fashion.
You'll need a proxy list to set this up. We picked up a few more from the Free Proxy List website.
Here's an example showing how to rotate proxies in Crawlee.
// import the required classes
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
// create a new ProxyConfiguration instance and specify your proxy URL
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://138.68.235.51:80',
'http://72.52.87.199:3128',
'http://209.97.150.167:8080'
],
});
const startUrls = ['https://httpbin.org/ip'];
// initialize a CheerioCrawler class
const crawler = new CheerioCrawler({
// pass in proxy configuration instance
proxyConfiguration,
// modify requestHandler to log page content
async requestHandler({ request, body }) {
console.log(`Scraping: ${request.url}`);
const jsonData = JSON.parse(body);
console.log(jsonData);
},
});
await crawler.run(startUrls);
If all goes well, you'll get a different IP address for each request. Below are the results for three runs:
Scraping: https://httpbin.org/ip
{ origin: '138.68.235.51' }
Scraping: https://httpbin.org/ip
{ origin: '72.52.87.199' }
Scraping: https://httpbin.org/ip
{ origin: '209.97.150.167' }
Well done!
You can also define a custom proxy function with the ProxyConfiguration class. This allows you to determine which proxy to use when certain conditions or factors are met.
Let's say, for example, you want to set a proxy for a specific subdomain. You can add some logic to use proxies only for that domain.
Here's an example:
// ...
// create a new ProxyConfiguration instance and specify your proxy URL
const proxyConfiguration = new ProxyConfiguration({
newUrlFunction: (sessionId, { request }) => {
if (request?.url.includes('httpbin.org')) {
// for httpbin use specified proxy
return 'http://72.52.87.199:3128';
}
// don't use any proxy for other domains
return null;
}
});
// ...
This code uses the newUrlFunction option to instruct the crawler only to use proxies if the domain includes httpbin.org.
Priority-Based Proxy Rotation in Crawlee
The ProxyConfiguration class also allows you to specify proxy tiers, which categorize proxies based on their quality or priority for specific domains.
To set this up in Crawlee, you'll need to define your tiered proxy URLs using the tieredProxyUrls option. Note that this option only works with a crawler instance.
Let's say we want to categorize our original proxy rotating script; your new code will look like this:
// import the required classes
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
// create a new ProxyConfiguration instance and specify your proxy URL
const proxyConfiguration = new ProxyConfiguration({
tieredProxyUrls: [
'http://138.68.235.51:80',
'http://72.52.87.199:3128',
'http://209.97.150.167:8080'
],
});
const startUrls = ['https://httpbin.org/ip'];
// initialize a CheerioCrawler class
const crawler = new CheerioCrawler({
// pass in proxy configuration instance
proxyConfiguration,
// modify requestHandler to log page content
async requestHandler({ request, body }) {
console.log(`Scraping: ${request.url}`);
const jsonData = JSON.parse(body);
console.log(jsonData);
},
});
await crawler.run(startUrls);
This crawler will use the first proxy URL and switch to the second if the first is blocked. If the second is also blocked, it'll switch to the third.
Here's the result:
WARN CheerioCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session...
read ECONNRESET
{"id":"Z4ksPBuAW2xZkJW","url":"https://httpbin.org/ip","retryCount":1}
Scraping: https://httpbin.org/ip
{ origin: '72.52.87.199' }
The output shows that the first proxy was blocked, and the crawler switched to the second proxy, which was successful.
Congratulations! You can now configure a Crawlee proxy according to your project needs.
Remember, free proxies are unreliable, and this is evident if we look at the example above; websites easily detect them. To avoid getting blocked, you need premium proxies; more on that in the next section.
Premium Proxies to Avoid Getting Blocked
Premium proxies can increase your chances of avoiding detection.
That being said, not all proxies work the same. Residential proxies are the best proxy type for web scraping since they use IP addresses assigned by actual ISPs and linked to real devices.
The best ones, such as the ZenRows Residential Proxy, will boost your success rate significantly and offer additional features such as proxy rotation and geo-location.
ZenRows is the most reliable residential proxy provider out there, with over 55 million IPs across 185+ countries. It also offers intelligent proxy selection, geo-targeting, enterprise-grade uptime, and so on.
Below is a quick guide on how to get started with ZenRows' Residential Proxies.
Sign up and navigate to the Proxy Generator. Your auto-generated residential proxy will be at the top of the page.
You can set your username and password by going to the credentials tab and filling in the appropriate fields.
Once you're all set up, copy your proxy URL and use it in your Crawlee crawler, just like in the previous example.
Proxy URL: 'http://<ZENROWS_PROXY_USERNAME>:<ZENROWS_PROXY_PASSWORD>@superproxy.zenrows.com:1337'
Here's the final code using ZenRows residential proxies:
// import the required classes
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
// create a new ProxyConfiguration instance and specify your proxy URL
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: ['http://<ZENROWS_PROXY_USERNAME>:<ZENROWS_PROXY_PASSWORD>@superproxy.zenrows.com:1337'],
});
const startUrls = ['https://httpbin.org/ip'];
// initialize a CheerioCrawler class
const crawler = new CheerioCrawler({
// pass in proxy configuration instance
proxyConfiguration,
// modify requestHandler to log page content
async requestHandler({ request, body }) {
console.log(`Scraping: ${request.url}`);
const jsonData = JSON.parse(body);
console.log(jsonData);
},
});
await crawler.run(startUrls);
Since ZenRows uses smart proxies that automatically rotate when necessary, you don't need to specify multiple proxies.
Here are the results for two runs:
Scraping: https://httpbin.org/ip
{ origin: '175.29.134.5' }
Scraping: https://httpbin.org/ip
{ origin: '74.136.196.173' }
Awesome, right? ZenRows makes web scraping easy.
Conclusion
Configuring a Crawlee proxy can help you avoid getting blocked by IP-based restrictions. However, websites can block a single proxy, so you must rotate between servers for a web scraping experience.
That said, never rely on free proxies, as they're unreliable. Opt for premium proxies such as ZenRows’ Residential Proxy for a consistent performance and high success rate.