Are websites blocking your attempts to extract data or uncover SEO insights using Screaming Frog?
Proxies can help you disguise your web crawling activities, preventing target servers from identifying and blocking your crawler.
In this guide, you'll learn how to set up a Screaming Frog proxy, and learn additional tips to enhance your web scraping or SEO strategy.
What Is Screaming Frog?
Screaming frog is a desktop app for Windows, macOS, or Linux that allows you to crawl websites for XML sitemaps, links, page titles, meta descriptions, and various data points that may be valuable for technical analysis.
SEO experts often use it for competitive monitoring and auditing websites for issues, such as broken links, outbound and in-bound links, missing metadata, etc.
Screaming Frog is free to crawl up to 500 URLs and is easy to use. You only need to insert the homepage of the website you want to crawl, and Screaming Frog will crawl and audit the entire website, similar to Google bot.
It also updates the crawl data in real time and provides information about the crawl (speed, total number of URLs completed and remaining, etc.) at the bottom of the dashboard.
You can also pause or save a crawl to resume at any time. For more details on how this tool works, check out the Screaming Frog user guide.
Before we dive in, download and install SEO Spider if you haven't already. Once that's set, get ready to configure a Screaming Frog proxy.
How to Add a Proxy Server in Screaming Frog SEO Spider
Screaming Frog's intuitive dashboard makes configuring the SEO Spider to use a proxy server easy. All you need to do is navigate to the proxy configuration window and add the proxy's address and port.
Here's a step by step guide.
Start by navigating to File > Settings > Proxy for Windows/Linux or Screaming Frog SEO Spider > Settings > Proxy for macOS.
You'll be directed to the configuration window. Tick the Use Proxy Server checkbox and enter your proxy settings.
For this tutorial, we'll grab a free proxy from the Free Proxy List website.
- Address: 18.188.127.31
- Port: 3128
Lastly, click OK and restart for the proxy configuration to take effect.
That's it, you can now route your crawler through the specified proxy server.
However, it's worth noting that free proxies are unreliable as websites can easily detect and block them. Consequently, the one used in this article may not work at the time of reading. You may need a new one from the Free Proxy List website.
If you're using a paid or premium proxy, you'll need authentication; more on that in the next section.
Proxy Authentication in Screaming Frog
Some proxy providers require authentication details, such as username and password, to grant access to their proxy servers.
To authenticate a Screaming Frog proxy, navigate to the configuration window as in the previous step (File > Settings > Proxy), tick the Use Proxy Credentials checkbox, and enter your authentication details.
That's it!
You can further enhance your crawler by switching between proxies. Let's explore that in the upcoming section.
Use Rotating Proxies to Avoid Getting Blocked in Screaming Frog
While configuring a Screaming Frog proxy is a great starting point, websites can still block your requests or ban your proxy.
Anti-bot systems often flag excessive requests from a single IP address as malicious activity, leading to IP bans or rate limiting errors.
To scrape without getting blocked, you must rotate between proxies. This enables you to distribute traffic across multiple servers, mimicking different users, as your requests appear to originate from unique devices.
When you rotate proxies, you can avoid IP bans, and bypass geo-restrictions.
Unfortunately, this SEO Spider doesn't natively support proxy rotation as you can only specify a single proxy server. To rotate proxies in Screaming Frog, you must use a dedicated premium proxy service with proxy auto-rotation features.
You'll need a service that offers advanced features, such as anti-bot bypass and geo-location for a smooth crawling process and access to localized content.
That's why residential proxies are the recommended proxy type for web scraping. The best ones, such as the ZenRows Residential Proxies, will significantly increase your success rate, allowing you to focus on your data extraction or SEO audit goals.
ZenRows automatically rotates premium residential proxies under the hood. Therefore, you only need to specify the proxy details in your Screaming Frog configuration, and ZenRows will automatically rotate between proxies during the crawl.
Some features of the ZenRows residential proxies, include:
- 55M+ IPs across 185+ countries.
- Geo-location.
- 99.9% uptime guarantee.
Below is a quick guide on how to get started with ZenRows' Residential Proxies.
Sign up and navigate to the Proxy Generator page. You'll find your auto-generated residential proxy at the top of the page.
You can customize the settings by navigating to the Credentials tab and setting a proxy username and password.
Once you're all set up, copy your proxy credentials for use in Screaming Frog, just as in the previous example.
- Address: superproxy.zenrows.com
- Port: 1337
- Username: <ZENROWS_PROXY_USERNAME>
- Password: <ZENROWS_PROXY_PASSWORD>
Navigate to the proxy configuration window, and enter your proxy address and port. Remember to include your ZenRows username and password.
Congratulations! You're now equipped to crawl any website using ZenRows and avoid geo-restrictions.
Conclusion
Proxies can disguise your web activities and grant you access to website data and SEO insights while using Screaming Frog. You can even enhance your crawler by rotating between proxies.
However, free proxies are unreliable and not all premium proxies offer the same features.
Since Screaming Frog only allows you to configure one proxy server, you need a provider that not only provides access to geo-restricted data but also automatically rotates proxies for you.
ZenRows, one of the top residential proxy providers, offers all this and more.
Sign up now to try ZenRows for free.