What is robots.txt?
robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may visit. It lives at a fixed address, such as https://example.com/robots.txt, and follows the Robots Exclusion Protocol, which the IETF published as RFC 9309 in 2022.
Every well-behaved crawler reads it before crawling a host. The file states the owner's wishes and relies on crawlers to follow them. Nothing technical stops a crawler that ignores it, which is why sites that care about enforcement also run anti-bot systems.
What a robots.txt file looks like
User-agent: *
Disallow: /cart/
Disallow: /search?
Allow: /search?q=
User-agent: Googlebot
Allow: /
Crawl-delay: 5
Sitemap: https://example.com/sitemap.xml
The directives
| Directive | What it does | Part of RFC 9309? |
|---|---|---|
User-agent |
Starts a group of rules for a named crawler, or * for all |
Yes |
Disallow |
A path prefix the crawler should not request | Yes |
Allow |
An exception inside a disallowed path | Yes |
Sitemap |
The location of an XML sitemap | Widely supported, outside the RFC |
Crawl-delay |
Seconds to wait between requests | Non-standard. Bing honours it, Google ignores it. See crawl delay |
How crawlers interpret the rules
- The most specific group wins. A crawler uses the group whose
User-agentmatches its own name most closely, and falls back to*only if none does. - The longest matching rule wins. If
Disallow: /searchandAllow: /search?q=both match, the longer path decides. - Wildcards.
*matches any sequence of characters and$anchors the end of the URL, soDisallow: /*.pdf$blocks PDFs. - Paths are case-sensitive.
/Admin/and/admin/are different rules. - A missing file means everything is allowed. A 404 on robots.txt is read as no restrictions. A server error is usually treated more cautiously.
robots.txt is not a security or privacy tool
Disallowing a path does not hide it. The file is public, so listing /internal-reports/ in it tells everyone the path exists. It also does not remove pages from search results: a disallowed page can still be indexed from links pointing at it. To keep a page out of an index, use a noindex meta tag or header, and to keep data private, use authentication.
Common mistakes
- Blocking CSS and JavaScript, which stops search engines rendering the page properly.
- A stray
Disallow: /left over from a staging site, which blocks the whole site. - Putting the file in a subdirectory. Crawlers only look at the root of each host, and each subdomain needs its own file.
- Treating it as a scraping permission slip. robots.txt says what the site owner prefers. Terms of service, copyright and data protection law still apply.
robots.txt in web scraping
For a scraper or crawler, reading robots.txt first is cheap and useful. It shows which areas the owner wants left alone, it often points at a sitemap that saves a lot of discovery, and Crawl-delay hints at a request rate the site can handle. The blog has a practical guide to robots.txt for web scraping.
For any crawler, the other useful signals sit on the pages themselves: their pagination, their links and how many items a page says it holds.
Frequently asked questions
Where is the robots.txt file located?
Always at the root of the host: https://example.com/robots.txt. A file anywhere else is ignored, and shop.example.com needs its own.
Is it illegal to ignore robots.txt?
robots.txt is a convention rather than a law in most places, but ignoring it can still matter: courts and regulators have looked at it as evidence of what a site owner permitted. Treat it as the site's stated wishes and check terms of service and local law for anything commercial.
Does robots.txt stop AI crawlers?
Only the ones that honour it. Many AI companies publish user agents, such as GPTBot or ClaudeBot, that site owners can disallow by name. Crawlers that do not identify themselves or ignore the file are handled by bot protection rather than robots.txt.
What is the difference between robots.txt and a meta robots tag?
robots.txt controls whether a crawler fetches a URL at all. A meta robots tag such as noindex sits inside a page and controls what a search engine does with it after fetching it.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026