Robots.txt Generator
Pick a starting point, add rules per crawler, then copy the result into a file named robots.txt at your site's root.
Points crawlers to your sitemap. Use the full absolute URL, not a relative path.
About this tool
A robots.txt file sits at the root of your domain (e.g. example.com/robots.txt) and tells well-behaved crawlers which parts of your site they may or may not request. It's a convention, not an access control: it doesn't password-protect anything, and a page you disallow can still get indexed if it's linked from elsewhere, just without its content crawled.
User-agent, Disallow, and Allow
Each block starts with a User-agent line naming a crawler (* matches all of them), followed by Disallow lines for paths that crawler shouldn't request and optional Allow lines that carve out exceptions within a disallowed path. An empty Disallow: means nothing is blocked for that agent.
Sitemap and crawl-delay
The Sitemap directive can point to your sitemap from any block and is read independently of the user-agent rules. Crawl-delay is supported by some crawlers (like Bing) but ignored by Google, which paces crawling based on its own signals instead, so it's left out of this generator's presets.
Keeping pages out of search results
If your goal is to stop a specific page from appearing in search results, a noindex meta tag or header is the correct tool, not robots.txt. Disallowing a page here can actually prevent search engines from seeing a noindex tag on it, since they never crawl the page to find it.