A robots.txt file sits at the root of your domain (for example https://example.com/robots.txt) and tells crawlers which URLs they may request. This tool writes the file for you in the format Google, Bing and other major crawlers expect, checks your paths for common mistakes, and lets you download the result ready to upload. Nothing you enter leaves your browser.
/. Google and Bing understand * as a wildcard and $ as end-of-URL, so /*.pdf$ blocks every PDF.Allow line can reopen a folder inside a blocked one. Google applies the most specific (longest) matching rule.https://. Our XML Sitemap Generator can create one.robots.txt to your site's root folder, usually public_html.Robots.txt controls crawling, not indexing. A blocked page can still appear in Google results, usually without a description, if other sites link to it. To keep a page out of search, let Google crawl it and add a noindex meta tag instead. Blocking the page in robots.txt would stop Google from ever seeing that tag.
It is also not a security measure. The file is public, and badly behaved bots ignore it. Never list secret folders in it expecting them to stay hidden; protect them with a password.
A few more limits worth knowing. Google reads only the first 500 KiB of the file. Each subdomain needs its own robots.txt, so shop.example.com does not inherit the rules of example.com. And Google ignores Crawl-delay; it adjusts its crawl rate automatically based on how your server responds.
Several AI companies publish crawler names that respect robots.txt. Disallowing GPTBot (OpenAI), ClaudeBot (Anthropic), meta-externalagent (Meta), CCBot (Common Crawl) and Bytespider (ByteDance) asks those crawlers not to collect your pages. Google-Extended and Applebot-Extended work differently: they are control tokens rather than crawlers, and they tell Google and Apple not to use pages already crawled by Googlebot or Applebot for AI model training. Blocking them does not remove you from Google Search or Apple's Siri and Spotlight results.
The list changes as companies launch new bots, so review it once or twice a year. Also note that user-triggered fetchers, such as an assistant opening a link a person pasted, may not follow robots.txt at all.