Robots.txt Generator for Search and AI Crawlers

robots.txt is a plain-text file at the root of a host that asks crawlers to stay away from some URLs. The format was standardized in 2022 as RFC 9309, the Robots Exclusion Protocol. A file is made of groups: one or more User-agent lines followed by Allow and Disallow rules.

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

Every crawler without a group of its own follows the * group. Rules match the start of the URL path, and the longest matching rule wins, so /admin/help/start is allowed while /admin/settings is blocked. The file applies only to the exact host, protocol and port it is served from: shop.example.com needs its own.

Two limits matter. robots.txt is a request, not a lock: the RFC says its rules "are not a form of access authorization", so protect private pages with a login. And it controls crawling, not indexing: Google can still list a blocked URL it finds through links, without a description. To keep a page out of results, let it be crawled and add a noindex robots meta tag.

Most AI companies now separate the crawler that gathers training data from the ones that power their search and answers, so you can opt out of one and keep the others. These are the training controls, from each operator's own documentation:

The Block AI training preset writes exactly this:

User-agent: *
Allow: /

# Opt out of these crawlers (see each operator's documentation)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Amazonbot
User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/sitemap.xml

Search and answer bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed, so you can still be found and cited. Anthropic, for example, documents a separate bot for each job: ClaudeBot for training, Claude-SearchBot for search and Claude-User for questions users ask. Note that Google-Extended and Applebot-Extended are control tokens, not crawlers: Google and Apple crawl with their usual bots and check the token before using your pages for AI. Fetchers that act for a user, like ChatGPT-User and Perplexity-User, may not follow robots.txt at all, OpenAI and Perplexity say. Microsoft has no robots.txt token for Bing's AI features; it uses the nocache and noarchive meta tags instead.

WordPress serves a virtual robots.txt that blocks the dashboard but keeps admin-ajax.php reachable, because themes and plugins call it from public pages. The WordPress preset matches it, with the core wp-sitemap.xml. If an SEO plugin writes your sitemap, point the Sitemap line at its URL instead.

For Next.js, never block /_next/: it holds the JavaScript and CSS Google needs to render your pages. Block API routes if you like, or generate the file from code:

// app/robots.ts
import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [{ userAgent: "*", allow: "/", disallow: "/api/" }],
    sitemap: "https://example.com/sitemap.xml",
  };
}

For a staging site, Disallow: / keeps polite crawlers out, but it doesn't stop indexing of URLs that leak through links. Put staging behind a password, or send an X-Robots-Tag: noindex header.

Neither. Google's parser reads only user-agent, allow, disallow and sitemap, and Google retired its handling of unsupported rules such as noindex on September 1, 2019. Other crawlers differ on Crawl-delay (seconds between requests):

If Googlebot is overloading your server, Google's advice is to return 500, 503 or 429 status codes for a few hours to a day or two, which makes it crawl more slowly; then go back to 200.

• A blank line doesn't end a group. Two User-agent lines separated by an empty line still share the rules after the second one. The generator writes Allow: / for a group with no rules so it can't merge by accident.
• **A named group replaces *.** Once User-agent: Googlebot has a group, Googlebot ignores everything in the * group. Repeat the rules it should keep.
• Paths must start with `/`. Disallow: admin never matches anything, and neither does a full URL.
• Prefixes match more than you think. Disallow: /blog also blocks /blog-tips. Use /blog/, or /blog$ for that exact URL.
• Blocking CSS and JavaScript stops Google rendering your pages properly.
• Size. Google reads only the first 500 KiB, the minimum RFC 9309 requires parsers to handle; rules after that are ignored.

Every one of these is flagged in Checks with the line to fix. To try a live file, use the robots.txt tester.