robots.txt: telling crawlers where not to go
robots.txt is a plain text file at the root of a website, such as example.com/robots.txt, that tells search engine crawlers and other bots which paths they should not request. It is one of the oldest web conventions, from 1994, and since 2022 an official internet standard, RFC 9309.
Fill in the user agent, the paths to allow and disallow, and your sitemap address, and this generator writes a valid robots.txt you can upload to the root of your site.
How to use it
- Keep User-agent as * to address all crawlers, or name one, such as Googlebot.
- List the paths crawlers should skip under Disallow, one per line, for example /admin/ or /cart.
- Add Allow paths for exceptions inside a disallowed folder.
- Enter your sitemap URL, then upload the file as robots.txt to the root of your domain.
Example: a small shop
User-agent: * Disallow: /admin/, /cart Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Disallow: /cart Sitemap: https://example.com/sitemap.xml
Paths are prefixes: /cart also matches /cart/checkout and /cartoon. Add a trailing slash to limit a rule to a folder.
robots.txt is not security
The file is a polite request, and it is public. Well-behaved crawlers such as Googlebot and Bingbot follow it; malicious bots ignore it, and anyone can read it to see which paths you would rather keep quiet. Protect private areas with authentication, never with robots.txt alone.
Blocking crawling is not the same as removing a page
A disallowed page can still appear in search results if other sites link to it; Google then shows the address without a description. To keep a page out of search results, let it be crawled and add a noindex meta tag or X-Robots-Tag header instead. Never block the CSS and JavaScript your pages need, or search engines cannot render them properly.
AI crawlers
Many AI companies publish crawler names that respect robots.txt, such as GPTBot from OpenAI, ClaudeBot from Anthropic and Google-Extended for Google's AI training. To opt out, add a separate group for each, with User-agent set to that name and Disallow: /. Check each company's documentation, since names and policies change.
Questions
Where must the file live?
At the root of each host, exactly at /robots.txt. A file in a subfolder is ignored, and www.example.com and shop.example.com each need their own.
What does an empty Disallow mean?
Disallow: with nothing after it allows everything. Disallow: / blocks the whole site.
How quickly do changes take effect?
Google generally caches robots.txt for up to 24 hours, so allow a day before checking the effect.