Where does robots.txt go?

At the root of the host it applies to, for example https://example.com/robots.txt. Each subdomain needs its own file, and a file in a subfolder is ignored.

Does Disallow remove a page from search results?

No. It stops compliant crawlers from fetching the page, but the URL can still be indexed if other pages link to it. To keep a page out of results, allow crawling and use a noindex meta tag or header, or protect the page with a login.

Do AI crawlers respect robots.txt?

The crawlers listed here are documented by their operators as honouring robots.txt. It is a request, not access control: a crawler that ignores the standard is not stopped by it.

ToolHub

Robots.txt Generator

Build a robots.txt file with allow and disallow rules per crawler, sitemap links and an optional block for AI crawlers. Copy or download the result.

Stable

Online

Start from a template

A template replaces the rule groups and keeps your sitemap URLs.

Rule group 1

User agent

Use * for all crawlers. Separate several names with commas.

Crawl delay in seconds (optional)

Honoured by Bing and Yandex. Google ignores it.

Disallow

One path per line, starting with /. A * matches any characters.

Allow

Exceptions inside a disallowed path, one per line.

Block AI crawlers

Adds a group that disallows the whole site for: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai.

Sitemap URLs

Full URLs, one per line.

robots.txt

User-agent: *
Disallow:

Save the file as robots.txt in the root of your site. Disallow stops crawling; it does not remove pages that are already indexed.

Create a robots.txt file without remembering the syntax. Start from a template for an open site, a blocked site, WordPress or an online shop, then add groups of rules for all crawlers or for named ones such as Googlebot or Bingbot. One switch adds a group that blocks crawlers collecting content for AI training and AI answers, including GPTBot, ClaudeBot, Google-Extended and CCBot. Paths that do not start with a slash or wildcard and sitemap entries that are not full URLs are reported and left out, so the file you download is valid. Everything runs in your browser.

Templates for open, blocked, WordPress and shop sites
Separate rule groups for all crawlers or named ones
One switch to block common AI crawlers
Sitemap links and optional crawl delay
Invalid paths and sitemap URLs are flagged and left out
Copy the file or download robots.txt