Build a correctly formatted robots.txt file with crawl rules, sitemap links, and optional AI crawler blocking, without needing to memorise the syntax.
Build a correctly formatted robots.txt file with crawl rules, sitemap links, and optional AI crawler blocking, without needing to memorise the syntax.
Upload the downloaded file to the root of your domain so it's reachable at yoursite.com/robots.txt, exactly that address, not inside any subfolder.
Robots.txt is a set of instructions crawlers can choose to respect, not a security or access control mechanism. Never rely on it to hide sensitive content, since the file itself is publicly visible and some bots ignore it entirely. All generation happens locally in your browser.
Found a technical issue?
Our technical SEO team fixes crawl errors, redirects, schema and indexing so Google can rank every page.
A robots.txt file is a plain text file at the root of your website that tells crawlers which parts of the site they may visit. Search engines like Google and Bing check it before crawling, and so do most AI crawlers.
This free robots.txt generator builds a correctly formatted file without you having to memorise the syntax. Set default rules for all bots, list paths to disallow such as admin or checkout pages, add your sitemap, set an optional crawl delay, and block AI crawlers like GPTBot, ClaudeBot and CCBot with a tick.
A single wrong line can block your whole site from Google, so a generator that gets the format right is worth using even if you know the basics.
A robots.txt file lives at yourdomain.com/robots.txt and uses User-agent, Disallow, Allow and Sitemap lines to guide crawlers. Use the generator above to build one, then upload it to your site's root folder or paste it into your SEO plugin's robots.txt editor.
Start from "allow everything", which is the right default for most websites, and only add the blocks you need.
Used to suggest your sitemap address.
Add folders crawlers should skip, such as /wp-admin/, /cart/ or internal search results.
Optional. Tick any AI bots you do not want collecting your content.
Save as robots.txt in your site root, or paste into Yoast or Rank Math's file editor.
Rules are grouped by user agent. Crawlers follow the most specific group that matches their name.
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /cart/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap_index.xml
Robots.txt controls crawling, not indexing. To keep a page out of Google, use a noindex meta tag and let Google crawl it.
A WooCommerce store tidies up its crawl budget.
Only a handful of directives exist. Support varies slightly between crawlers.
| Directive | What it does | Supported by |
|---|---|---|
| User-agent | Names the crawler a group of rules applies to | All major crawlers |
| Disallow | Blocks crawling of a path | All major crawlers |
| Allow | Re-opens a path inside a blocked folder | Google, Bing and most others |
| Sitemap | Points crawlers to your XML sitemap | Google, Bing and most others |
| Crawl-delay | Seconds to wait between requests | Bing and some others, ignored by Google |
| * and $ wildcards | Match any characters, or the end of a URL | Google and Bing |
Less is usually more. Most sites need only a few lines.
In the root folder of your domain so it loads at yourdomain.com/robots.txt. Each subdomain needs its own file.
No. It stops crawling, not indexing. Use a noindex tag, or remove the page and return a 404 or 410, to take it out of results.
Add a group for each bot, such as User-agent: GPTBot with Disallow: /. The generator lists the common AI crawlers so you can tick the ones to block.
No. Google ignores crawl-delay. Bing and some smaller crawlers respect it.
It is not required, but it is good practice so you can point to your sitemap and keep crawlers out of low-value areas.
Yoast SEO and Rank Math both include a robots.txt editor in their tools settings, so you can paste the generated rules without FTP.
Our technical SEO audits check robots.txt, sitemaps, indexing and crawl budget, then fix what is holding your site back. Explore our technical SEO services.