What is a 401 error?

A 401 Unauthorized means the server requires authentication for this resource and did not receive valid credentials. The page is there. You have not proved you are allowed to read it.

The name is a misnomer that causes real confusion: 401 is about authentication, meaning who you are, while 403 Forbidden is about authorization, meaning what you are allowed to do once you are known. A 401 says "identify yourself". A 403 says "I know who you are and the answer is still no".

How to recognise a real 401

A properly implemented 401 includes a WWW-Authenticate header naming the scheme the server expects, such as Basic, Bearer or Negotiate. That header is the fastest way to know what the server actually wants, and its absence is a hint that the 401 is being used loosely, sometimes by a bot filter that is really refusing you on identity.

What causes it during scraping

• No credentials sent. The obvious case: the resource is behind a login or an API key you did not supply.
• A session that expired. Long crawls outlive cookies. A run that starts fine and turns into a wall of 401s partway through has usually lost its session rather than been blocked.
• Credentials sent to the wrong place. An Authorization header authenticates you to the destination server. Proxy-Authorization authenticates you to a proxy. Swapping them produces a 407 or a 401 depending on which end was expecting what.
• A token that expired mid-run. Bearer tokens usually carry a short lifetime, and a crawl longer than that lifetime needs a refresh step rather than a single token fetched at the start.
• Cookies not persisted across requests. Each request being treated as a fresh visitor means the session established at login is never used again.

Keeping a session alive across a crawl

The practical fix for most scraping 401s is state. Authenticate once, keep the cookies, and reuse them, which also means keeping the same IP for the duration where the site ties a session to an address. A session that hops between IPs on every request will be invalidated by many sites even when the cookies are correct.

Where Zenrows fits

Zenrows Fetch takes custom headers on the request, so an Authorization header or a cookie captured at login travels with each call. For sites that bind a session to an address, the session ID parameter keeps the same IP across a sequence of requests, which is what stops an authenticated crawl from being logged out halfway through. Where the login itself needs a real browser, Browser Sessions drives it with Puppeteer or Playwright and carries the resulting cookies.