HTTP errors in web scraping
A failed scrape almost always arrives as a status code. Some of them mean the site rejected you, some mean a proxy in the chain gave up, and some mean the origin server itself is down. Telling them apart matters, because the fix is completely different: a 407 is a credentials problem you can fix in one line, and a 503 may mean backing off for an hour. These pages cover the codes that show up in scraping specifically, not the ones you only meet when building your own API.
What do HTTP status codes mean in web scraping?
A status code tells you which part of the chain rejected your request, the target site, a proxy in between, or the origin server, and each one calls for a different fix.
What is a 200 status code?
A 200 means the server accepted the request and returned a body, though in scraping that body can still be a block page rather than the content you asked for.
What is a 401 error?
A 401 means the server needs credentials it did not get, or got credentials it rejected, so the page exists but you are not authenticated to see it.
What is a 406 error?
A 406 means the server could not produce a response matching the Accept headers you sent, and in scraping it is often triggered by headers that do not look like a real browser's.
What is a 407 error in web scraping?
A 407 means the proxy you are routing through rejected your credentials or received none, so the request never reached the target site at all.
What is a 408 error?
A 408 means the server gave up waiting for your request to finish arriving, so the problem is on the way in rather than in the page you asked for.
What is a 410 error?
A 410 means the page has been deliberately removed and is not coming back, which makes it more useful than a 404 when you are maintaining a crawl.
What is a 413 error in web scraping?
A 413 means a body exceeded a size limit, and in scraping it usually comes from your own API rejecting an oversized response rather than from the site you were reading.
What is a 444 status code?
A 444 is an nginx-only code meaning the server closed the connection without sending any response at all, and it is frequently used to drop traffic a site has decided is automated.
What is a 451 error?
A 451 means the content is being withheld for legal reasons, most often geographic, so the fix is requesting from a region where the page is served rather than changing anything about your client.
What is a 500 error?
A 500 means the server hit an unhandled error while processing your request, and in scraping it is worth checking whether something in your request caused it before assuming the site is broken.
What is a 502 error?
A 502 means a server acting as a gateway got an invalid response from the server behind it, so the failure is somewhere in the chain rather than at either end you control.
What is a 503 error in web scraping?
A 503 means the server cannot handle the request right now. In scraping it is usually temporary overload or maintenance, but it is also what several anti-bot systems return instead of a 403.
What is a 504 error?
A 504 means a gateway waited for an upstream server and gave up, so something in the chain was too slow rather than broken.
What is a 520 error?
A 520 is a Cloudflare-specific code meaning it reached the origin server but got back a response it could not interpret, which makes it the vaguest error in the Cloudflare range.
What is a 522 error?
A 522 means Cloudflare could not establish a connection to the origin server at all, so the site's own infrastructure is unreachable and nothing about your request is involved.
Get reliable web data in minutes.
Free plan, 5,000 credits every month, no credit card required.