What do HTTP status codes mean in web scraping?

TL;DR

Every failed scrape arrives as a status code, and the code tells you who rejected you. The 4xx family means the request itself was refused, usually on identity or credentials. The 5xx family means something broke after the request was accepted, either at a proxy or at the origin. Reading the family first, then the specific code, is faster than guessing, because the fix for a 407 takes one line and the fix for a 503 may be an hour of waiting.

Reading the code before reacting

Three questions, in order:

1. Who answered? A 407 comes from your proxy. A 502 or 504 comes from a gateway between you and the site. A 403 or 401 comes from the site. The response headers usually name the responder.
2. Is it about you or about them? 4xx says the request was rejected. 5xx says the server failed. Retrying helps the second far more than the first.
3. Does a browser see the same thing? If a page loads by hand while your client gets errors, the code describes your request rather than the site's health.

The codes a scraper actually meets

Not every status code shows up in scraping. These are the ones that do.

Cloudflare also issues its own 1xxx error numbers on the block page rather than in the status line. Those are covered on the blog, starting with error 1015.

The trap worth knowing

A successful status code does not guarantee useful content. Many anti-bot systems return 200 with a challenge or interstitial page, so a pipeline that only checks status == 200 will happily store thousands of block pages and report success. Check the body as well as the code, at minimum for expected content on the page.

Key takeaways

Read the family before the number: 4xx means your request was refused, 5xx means something failed after it was accepted. Identify who answered, since a proxy error and a site error look similar in logs and have nothing in common as problems. And do not trust a 200 on its own, because a block page can carry one.