What is a 200 status code?

A 200 OK means the server understood the request, accepted it, and returned a response body. It is the code every request is aiming for, and in most of the web it means exactly what you would expect.

In scraping it deserves more suspicion than that.

Why a 200 can still be a failure

Anti-bot systems have a choice about how to refuse you. Returning a 403 is honest and easy to detect, which is precisely why many of them do something else: they return 200 with a challenge page, an interstitial, or a stripped-down version of the site. From your client's point of view the request succeeded. From your pipeline's point of view you now have a stored page containing no data.

This is the most common cause of a scraper that reports a high success rate and produces an empty dataset. Checking response.status == 200 and moving on is not enough.

What to check instead

A content assertion catches this where a status check cannot:

• Expect something specific. A product page should contain a price element, an article should contain a headline. If the selector or field you need is missing, treat the response as failed regardless of its code.
• Watch the response size. Block pages are small and unusually consistent. A sudden run of responses all within a few hundred bytes of each other is worth alerting on.
• Look for challenge markers. Phrases such as "checking your browser", "verify you are human", or a vendor's script tag are reliable tells.
• Compare against a known-good sample. Storing one correct page per site and diffing structure against it catches soft blocks and site redesigns at the same time.

Related success codes

The 2xx family has a few others worth recognising during a crawl. A 204 means success with no body, which is normal for tracking endpoints and useless to scrape. A 206 means partial content and shows up when a range request was made, usually while downloading a file. A 201 means something was created, which you should almost never see while reading pages and which may indicate you have sent a POST by accident.

Key takeaways

A 200 confirms the server answered, and nothing more. In scraping specifically, treat it as a necessary condition rather than a sufficient one: assert on the content you expected to find, because a soft block arrives wearing the same status code as a success.