Guides · HolicAudit

Robots.txt: syntax, examples and mistakes to avoid

Published 2 November 2026

In short: robots.txt is a plain-text file at the root of your site that tells crawlers which paths not to fetch. It controls crawling, not indexing, and one wrong line can hide an entire site from search.

The syntax in one minute

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

User-agent starts a group. * means every crawler without its own group.
Disallow blocks paths that start with the value. An empty Disallow blocks nothing.
Allow makes an exception inside a blocked path.
* matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDFs.
Sitemap takes a full URL and can appear anywhere in the file.

Paths are case-sensitive. A crawler uses only the most specific group that names it, so if you add a Googlebot group, Googlebot ignores the * group completely. Repeat any shared rules.

Good starting points

Most small sites: User-agent: *, Allow: / and a Sitemap line. Nothing else. WordPress: the default virtual file blocks /wp-admin/ but allows /wp-admin/admin-ajax.php; keep that. Don't block /wp-content/, since Google needs your CSS and JavaScript to render pages. Online stores: block internal search results and endless filter combinations (for example Disallow: /*?*sort=) so crawlers spend their time on products.

Mistakes we see in audits

Disallow: / left over from a staging site after launch.
Blocking CSS, JavaScript or image folders, which stops Google rendering the page properly.
Using Disallow to remove pages from Google. Blocked pages can still be indexed from links; use noindex on a crawlable page instead.
Listing secret admin paths. robots.txt is public; protect those with authentication.
An HTML page or a 5xx error at /robots.txt. A server error makes Google pause crawling.

Test before you deploy

Paste a URL into our robots.txt checker to see which rule applies to Googlebot or any other crawler, and use the generator there for a clean starting file.

Frequently asked questions

Is robots.txt required?

No. Without one, crawlers may fetch everything. Adding a short file with a Sitemap line is still worthwhile.

Can robots.txt block AI crawlers?

Well-behaved ones, yes: GPTBot, CCBot, Google-Extended, ClaudeBot and others publish their user-agent names. It doesn't stop crawlers that ignore the file.