Robots.txt Generator

Build a correctly formatted robots.txt file. Choose the default rule, add paths to block or allow, add your sitemap, and optionally opt out of AI-training crawlers.

Adds GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Bytespider. Google Search and Bing indexing are not affected.

Bing and Yandex read this; Google ignores it.

Your robots.txt

    About the Robots.txt Generator

    A robots.txt file sits at the root of your domain (for example https://example.com/robots.txt) and tells crawlers which URLs they may request. This tool writes the file for you in the format Google, Bing and other major crawlers expect, checks your paths for common mistakes, and lets you download the result ready to upload. Nothing you enter leaves your browser.

    How to Use It

    1. Pick a default. Almost every public site should allow everything and block only specific areas. "Block everything" is meant for staging or private sites.
    2. List paths to block. Start each path with /. Google and Bing understand * as a wildcard and $ as end-of-URL, so /*.pdf$ blocks every PDF.
    3. Add exceptions. An Allow line can reopen a folder inside a blocked one. Google applies the most specific (longest) matching rule.
    4. Add your sitemap. It must be a full URL including https://. Our XML Sitemap Generator can create one.
    5. Generate, then upload the file as robots.txt to your site's root folder, usually public_html.

    What Robots.txt Does Not Do

    Robots.txt controls crawling, not indexing. A blocked page can still appear in Google results, usually without a description, if other sites link to it. To keep a page out of search, let Google crawl it and add a noindex meta tag instead. Blocking the page in robots.txt would stop Google from ever seeing that tag.

    It is also not a security measure. The file is public, and badly behaved bots ignore it. Never list secret folders in it expecting them to stay hidden; protect them with a password.

    A few more limits worth knowing. Google reads only the first 500 KiB of the file. Each subdomain needs its own robots.txt, so shop.example.com does not inherit the rules of example.com. And Google ignores Crawl-delay; it adjusts its crawl rate automatically based on how your server responds.

    Opting Out of AI Training

    Several AI companies publish crawler names that respect robots.txt. Disallowing GPTBot (OpenAI), ClaudeBot (Anthropic), meta-externalagent (Meta), CCBot (Common Crawl) and Bytespider (ByteDance) asks those crawlers not to collect your pages. Google-Extended and Applebot-Extended work differently: they are control tokens rather than crawlers, and they tell Google and Apple not to use pages already crawled by Googlebot or Applebot for AI model training. Blocking them does not remove you from Google Search or Apple's Siri and Spotlight results.

    The list changes as companies launch new bots, so review it once or twice a year. Also note that user-triggered fetchers, such as an assistant opening a link a person pasted, may not follow robots.txt at all.