What is a 500 error?

A 500 Internal Server Error is the catch-all: the server encountered a condition it did not know how to handle and gave up. It carries no information about what went wrong, deliberately, since exposing the stack trace would be a security problem.

The instinct is to treat it as somebody else's outage and retry. That is right about half the time.

When your request is the cause

A 500 is generated by application code, which means it can be triggered by input that code did not expect. During scraping, the usual triggers are:

• Malformed query parameters. A page size or offset outside the range the application handles, or a filter value it never anticipated.
• Unescaped characters in a URL. Especially in path segments assembled from scraped text.
• A URL built from a pattern that does not apply everywhere. Guessing ?page=2 on a site that paginates differently can reach a code path nobody tests.
• Missing headers the application assumes. Some applications assume a Referer or a session cookie and throw when it is absent.

The tell is consistency. A 500 that reproduces every time for one URL and never for others is about that URL. A 500 that appears across many URLs at once, then clears, is the site.

When it is genuinely theirs

Deploys, database failures and traffic spikes all surface as 500s, and they clear on their own. Retry with exponential backoff and jitter, and cap the attempts, since a server throwing 500s under load will not be helped by your retries.

Watch for the rate rising with your concurrency. If 500s appear only when you push parallelism up, you are contributing to the condition, and backing off is both the fix and the courteous thing to do.

How it differs from its neighbours

A 502 and a 504 come from a gateway in front of the application, reporting that something behind it misbehaved or did not answer. A 503 says the server knows it cannot serve you right now, which is a deliberate and usually temporary state. A 500 means the application actually ran and threw, which is why it is the one most likely to have been caused by what you sent.