FAHAQ

Robots.txt Generator

Build a valid robots.txt from simple options.

Group 1
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Free robots.txt generator. Add user-agent groups with allow and disallow paths, an optional crawl-delay and a sitemap URL, and get a valid robots.txt file to copy. Runs entirely in your browser.

How to use the Robots.txt Generator

  1. 01Set the user-agent (use * for all crawlers).
  2. 02Add the paths to disallow and allow, one per line.
  3. 03Optionally set a crawl-delay and add more groups.
  4. 04Add your sitemap URL, then copy the generated robots.txt.

Frequently asked questions

Where does the robots.txt file go?

It must live at the root of your domain, e.g. https://example.com/robots.txt. Crawlers look for it there and nowhere else.

Does Disallow guarantee a page won't be indexed?

No. Disallow blocks crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of the index, allow crawling and use a noindex meta tag.