Web Scraping Blog

Handle the User Agent in NodeJS

Learn how to set and rotate User Agents in NodeJS to successfully web scrape without getting blocked. You'll also find a handy tool to scale up.

Rubén del Campo

January 29, 2025 · 4 min read

How to Build a Web Crawler in Python

Learn to build a scalable Python web crawler. Manage millions of URLs with Boolm Filters, optimize speed with multi-threading, and bypass advanced anti-bots.

Idowu Omisola

March 5, 2026 · 10 min read

CloudProxy Tutorial to Scrape Websites in 2026

Learn how to use CloudProxy to use proxies for bypassing anti-bot measures. Also, find a more reliable CloudProxy alternative to scrape any website.

July 12, 2023 · 5 min read

CloudUnflare for Real IP Address Reconnaissance

Discover how you can use CloudUnflare to bypass Cloudflare and find a more effective alternative for your scraping project.

July 10, 2023 · 3 min read

cURL User Agent: How to Set or Change It

Learn how to set, change, and randomize your cURL User Agent to bypass anti-bot measures and scrape any data you want.

Yuvraj Chandra

January 26, 2025 · 4 min read

10 Web Scraping Challenges You Should Know

Discover the most common web scraping challenges you're likely to encounter and how to resolve them quickly and effectively.

July 4, 2023 · 6 min read

How to Use a HttpClient Proxy in C# (2026)

Learn how to use a proxy with HttpClient to avoid IP blocks and scrape in C# quickly and effectively.

July 3, 2023 · 8 min read

2 Easy Ways to Bypass "Please Verify You Are a Human"

Discover the most efficient method to bypass the "Please verify you are human" challenge by PerimeterX and resume your scraping efforts.

Yuvraj Chandra

December 5, 2025 · 3 min read

How to Bypass Cloudflare in PHP (2 Methods)

Learn two effective methods to bypass Cloudflare in PHP to scrape any website and get the data you need.

Idowu Omisola

September 19, 2024 · 4 min read

WhatWaf for Scraping Websites (2026)

Find out how you can use Whatwaf to bypass firewalls when web scraping and a better alternative for your next scraping project.

June 22, 2023 · 3 min read

Change the Selenium User Agent: Steps & Best Practices

Learn how to customize and rotate the Selenium User Agent when web scraping and find a solution to make the process easier.

Sergio Nonide

January 27, 2025 · 7 min read

Rust Web Scraping in 2026

Get started with web scraping in Rust following this step-by-step tutorial! Learn how to scrape a site and how to tackle the main challenges.

May 15, 2024 · 8 min read

Cloudflare 403 Forbidden: Bypass This Error

Learn how to bypass the Cloudflare 403 Forbidden Error and ensure the success of your scraping project.

Yuvraj Chandra

December 10, 2025 · 4 min read

CloudFail to Scrape Cloudflare Sites in 2026

Learn how to use CloudFail to bypass Cloudflare's anti-bot measures, as well as discover a more reliable alternative tool.

June 8, 2023 · 4 min read

CrimeFlare for Scraping Cloudflare Sites [+Alternatives]

Find all you need to know about CrimeFlare to scrape Cloudflare anti-bot measures, as well as alternatives for bypassing the protections.

April 9, 2025 · 4 min read

How to Use a Proxy With Selenium in Python

Learn to use a proxy with Selenium in Python to avoid being blocked while web scraping. This tutorial covers authentication, rotating proxies and more.

June 27, 2024 · 12 min read

How to Use Cloudscraper in Python & Fix Common Errors

Learn to use Cloudscraper in Python to bypass Cloudflare and scrape websites. Discover key features, proxies, and alternatives.

Idowu Omisola

September 19, 2024 · 7 min read

How to Use a Playwright Proxy

Learn to use Playwright proxies in 2025 for web scraping. Avoid IP bans, access geo-restricted content, and master proxy rotation and authentication.

Rubén del Campo

January 6, 2025 · 3 min read