Free tool · No sign-up

Robots.txt checker and generator

Updated 2 November 2026

Enter a page address to fetch that site's robots.txt, test whether Googlebot (or any crawler you name) may crawl the page, and spot rules that block your whole site. Below the checker is a generator for a clean starter file.

Robots.txt generator

How the test works

Crawlers read /robots.txt before anything else. The file is split into groups, each starting with one or more User-agent lines followed by Allow and Disallow rules. A crawler uses only the group with the most specific user-agent that matches its name, and falls back to the * group. Inside that group, the longest matching rule wins, and when an Allow and a Disallow rule are the same length, Allow wins. This tool follows those rules, which Google documents and which RFC 9309 standardised in 2022, including the * wildcard and the $ end-of-URL anchor.

Mistakes it catches

• A site-wide block. Disallow: / under User-agent: * is usually copied from a staging site and quietly removes a whole site from search.
• robots.txt that's really a web page. Single-page apps often answer every path, including /robots.txt, with their HTML. Crawlers can't read that.
• Server errors. If robots.txt returns a 5xx error, Google treats the site as fully disallowed for a while.
• No Sitemap line. A Sitemap: line with the full URL helps every crawler find your pages, not just the ones you've submitted to.
• Unsupported fields. Noindex: in robots.txt hasn't worked in Google since 2019; Crawl-delay is ignored by Googlebot.

Blocking is not hiding

Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of Google, allow crawling and add a noindex robots meta tag, or protect it with a login. robots.txt is public, so never list private paths you'd rather nobody knew about.

AI crawlers

You can opt out of some AI training crawlers, such as GPTBot, CCBot and Google-Extended, with their own groups. Blocking Google-Extended doesn't affect Google Search rankings. The generator below adds those groups if you tick the box.

Frequently asked questions