What is a 410 error?

A 410 Gone means the resource used to exist at this URL, has been intentionally removed, and the server does not expect it to return. It is a stronger statement than a 404, which only says the server cannot find anything here right now and leaves open the possibility that it will later.

Most sites never bother with the distinction and return 404 for everything. When a site does take the trouble to send 410, it is telling you something worth acting on.

Why it matters during a crawl

A crawl that revisits URLs needs a policy for dead ones, and the two codes deserve different policies.

A 404 could be a deployment glitch, a temporary database problem, or a page that will be restored. Dropping the URL on the first 404 loses content that comes back. The usual approach is to keep retrying on a decaying schedule and only give up after several consecutive failures.

A 410 is the site telling you to stop asking. Removing the URL from the frontier immediately is correct, and continuing to poll it wastes budget on both sides. On a large recurring crawl this is a real saving, and it also keeps you from looking like a scraper that ignores what it is told.

Where it shows up

Retail sites use 410 for delisted products, job boards use it for expired postings, and news sites occasionally use it for content pulled for legal reasons, though 451 is the more precise code for that. Any site with a genuine content lifecycle is a candidate.

The caching difference

There is a practical consequence beyond crawl hygiene: a 410 response is cacheable by default, while a 404 is not. Intermediate caches and your own layer will happily hold on to a 410, which is another reason not to keep requesting one. If you cache responses in your pipeline, honouring that default gives you the correct behaviour for free.