robots.txt Explained: Syntax, Examples & Mistakes

What Is robots.txt?

robots.txt is a plain-text file that lives at the root of your domain, for example https://example.com/robots.txt. It follows the Robots Exclusion Protocol, and it tells automated crawlers which parts of your site they are allowed to request. When a well-behaved bot like Googlebot visits, it reads this file first.

The single most important thing to understand: robots.txt controls crawling, not indexing. It asks crawlers not to fetch certain URLs. It does not guarantee those URLs stay out of Google's index. We will come back to that critical distinction below.

The Core Directives

A robots.txt file is made of groups. Each group starts with one or more User-agent lines followed by rules.

A note on Crawl-delay: Google does not obey it. You control Googlebot's crawl rate in Search Console instead. Bing and some other engines do honor it.

Working Examples

Allow everything (the default behavior, stated explicitly):

User-agent: *
Disallow:

Block a single folder but allow one file inside it:

User-agent: *
Disallow: /private/
Allow: /private/public-page.html

Sitemap: https://example.com/sitemap.xml

Give different rules to different bots:

User-agent: Googlebot
Disallow: /no-google/

User-agent: *
Disallow: /admin/

Paths are case-sensitive and matched from the start of the URL path. You can use * as a wildcard and $ to anchor the end of a URL.

robots.txt vs noindex

This is where most sites go wrong. The two mechanisms solve different problems:

• robots.txt Disallow stops crawlers from fetching a URL. If other sites link to that URL, Google can still index it as a bare, snippet-less result, because it never crawled the page to see a noindex tag.
• noindex (a meta robots tag or X-Robots-Tag HTTP header) reliably keeps a page out of the index, but the page must be crawlable for Google to see the tag.

The takeaway: if you want a page gone from search results, use noindex and do not block it in robots.txt. Blocking it prevents Google from ever seeing the noindex directive.

Common Mistakes

1. Blocking your entire site. A stray Disallow: / under User-agent: * hides everything. This happens constantly when a staging robots.txt is pushed to production.
2. Blocking CSS and JavaScript. Google renders pages like a browser. If you disallow /assets/ or /js/, Google may render a broken page and misjudge its quality and mobile-friendliness.
3. Using robots.txt to hide sensitive data. The file is public, and a Disallow line advertises exactly where your secrets live. Use authentication instead.
4. Expecting Disallow to remove a page from search results. As explained above, use noindex for that.
5. Relative Sitemap URLs. The Sitemap directive requires a full absolute URL.

How to Build and Test Yours

Start by generating a clean file, then verify it does what you expect before deploying:

1. Create the file with a robots.txt Generator so the syntax and groups are correct.
2. Add an absolute Sitemap line pointing to your XML sitemap.
3. Check specific URLs against your rules with a robots.txt Tester to confirm important pages are crawlable and the right paths are blocked.
4. Upload it to your domain root and re-test in Search Console.

Conclusion

robots.txt is a small file with an outsized impact. Used well, it guides crawlers and points them to your sitemap. Used carelessly, a single line can wipe your site from Google. Remember that it manages crawling rather than indexing, never block your CSS and JavaScript, and always test before you deploy.