FAHAQ

Robots.txt Generator

Build a valid robots.txt from simple options.

Group 1
robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
ShareXFacebookLinkedInWhatsAppRedditEmail

Free robots.txt generator. Add user-agent groups with allow and disallow paths, an optional crawl-delay and a sitemap URL, and get a valid robots.txt file to copy. Runs entirely in your browser.

What robots.txt does

robots.txt is a plain-text file at the root of your domain (https://example.com/robots.txt) that tells crawlers which paths they may or may not request. It's the first file most bots fetch. Used correctly it saves crawl budget and keeps low-value URLs (search results, cart pages, admin) out of the crawl queue.

Important: Disallow controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.

The core directives

A robots.txt file is made of groups. Each group starts with a User-agent line and lists rules:

Directive Meaning
User-agent Which crawler the rules apply to (* = all)
Disallow Path prefix the crawler must not request
Allow Exception that overrides a broader Disallow
Sitemap Absolute URL of your XML sitemap

Paths are case-sensitive and relative to the root. Disallow: / blocks the whole site; an empty Disallow: allows everything.

Common mistakes

  • Blocking CSS/JS that Google needs to render the page — this can hurt rankings.
  • Using robots.txt to hide a page from search (use noindex instead).
  • Forgetting the Sitemap: line — always use the absolute URL.
  • Placing the file anywhere other than the domain root; /subfolder/robots.txt is ignored.
  • Assuming it's a security control — the file is public and malicious bots ignore it.

Example

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public-guide.html

Sitemap: https://example.com/sitemap.xml

This blocks all bots from /admin/ and /cart/, carves out one public admin page, and points crawlers to the sitemap.

How to use the Robots.txt Generator

  1. 01Set the user-agent (use * for all crawlers).
  2. 02Add the paths to disallow and allow, one per line.
  3. 03Optionally set a crawl-delay and add more groups.
  4. 04Add your sitemap URL, then copy the generated robots.txt.

Frequently asked questions

Where does the robots.txt file go?

It must live at the root of your domain, e.g. https://example.com/robots.txt. Crawlers look for it there and nowhere else.

Does Disallow guarantee a page won't be indexed?

No. Disallow blocks crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of the index, allow crawling and use a noindex meta tag.