Robots.txt Generator

Build a robots.txt file with explicit AI-bot controls for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and CCBot. Most generators forget these.

Free forever. No signup. Runs in your browser.

Configuration

Allow search bots to crawl everything by default

Disallow paths (one per line)

Sitemap URL

Crawl delay (seconds, 0 = none)

AI / LLM training bots

Allow = your content can appear in AI answers. Disallow = you opt out of training and citation.

Allow GPTBot / ChatGPT-User (OpenAI)Allow ClaudeBot / anthropic-ai (Anthropic)Allow Google-Extended (Gemini, Bard, AI Overviews training)Allow PerplexityBotAllow CCBot (Common Crawl)

Output

# Generated by TheProjectSEO robots.txt tool
# https://theprojectseo.com/tools/robots-txt-generator

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /_next/

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt at your site root (e.g. https://example.com/robots.txt).