UtilibyWritten by the Utiliby editorial team · Last updated October 14, 2026 · Rules and formulas reviewed October 2026
What this free robots.txt generator does
Every crawler that visits your site, from Googlebot to AI bots, first asks one question: what am I allowed to fetch? The answer lives in a small text file called robots.txt. Get it right and crawlers spend their time on the pages that matter. Get it wrong and you can hide your whole site from search by accident.
This free online robots.txt generator builds the file for you. Start from a preset for WordPress, WooCommerce, an online shop, a blog or a staging site, add rules for specific crawlers, tick the AI crawlers you want to block, add your sitemap and copy the finished file. The built-in tester then checks real paths against your rules and shows which rule decided each result. It also warns about the mistakes that cause the most damage. It is free, private and needs no sign-up.
How to create a robots.txt file in five steps
- Pick a starting point. Choose Allow everything for a simple site, or Custom rules and press a preset such as WordPress.
- Add your rules. List paths to block under Disallow and exceptions under Allow. Add groups for other crawlers if needed.
- Add your sitemap. Enter the full sitemap URL.
- Test it. Enter important URLs in the tester and check that they are allowed, and that private ones are blocked.
- Upload it. Save the file as robots.txt, upload it to the root of your domain and open it in a browser to confirm.
Robots.txt syntax in plain English
| Line | Meaning |
|---|---|
| User-agent: * | The rules below apply to all crawlers |
| User-agent: Googlebot | The rules below apply only to Googlebot |
| Disallow: /private/ | Do not crawl anything under /private/ |
| Disallow: | Nothing is blocked |
| Disallow: / | Block the whole site |
| Allow: /private/help.html | Exception to a Disallow rule |
| Disallow: /*.pdf$ | Block URLs that end in .pdf |
| Sitemap: https://example.com/sitemap.xml | Where to find your sitemap |
Paths are case sensitive, start with a slash, and each group of rules begins with one or more user-agent lines. When two rules match, the longer, more specific path wins, and an Allow beats a Disallow of the same length.
Blocking AI crawlers
Many website owners now want a say in whether their content trains AI models. Robots.txt is the standard way to ask. Each AI company publishes a crawler name, such as GPTBot, ClaudeBot, CCBot or Google-Extended, and says it follows robots.txt. You block one by adding a group with its name and Disallow: /. This tool lists common ones with a tick box.
Two cautions. First, robots.txt is voluntary, so a crawler that ignores it is not stopped, and content already collected is not removed. Second, blocking a search crawler removes you from that search engine, while some names, such as Google-Extended, control only AI use and do not affect Google Search. Names and policies change, so check each company's documentation from time to time.
Common setups
- WordPress: block /wp-admin/, allow /wp-admin/admin-ajax.php and block internal search pages.
- WooCommerce: also block cart, checkout, account pages and filter parameters that create endless URL combinations.
- Online shops: block cart, checkout and account pages, and sorting or filter parameters that duplicate category pages.
- Staging sites: block everything, and also protect the site with a password because robots.txt is only a request.
- Blogs and brochure sites: usually allow everything and list the sitemap.
Common robots.txt mistakes
- Disallow: / on a live site, which blocks all crawling.
- Blocking CSS and JavaScript, which stops Google from rendering pages.
- Using robots.txt to hide pages from search. A blocked URL can still be listed. Use noindex instead.
- Combining noindex with a block. Crawlers cannot see the noindex tag on a page they may not fetch.
- Putting the file in the wrong place. It must be at the root of each subdomain.
- Relative sitemap URLs. Use the full address.
- Using it as security. The file is public, and anyone can read it.
Testing before you publish
The tester reads the file above, or any robots.txt you paste, finds the group for the crawler you pick and checks each path with the same longest-match rule that Google describes. It shows whether the path is allowed or blocked and which line decided it. Test your home page, key category pages and a few articles, and also test the private paths you meant to block. After you upload the file, you can also check it with the robots.txt report in Google Search Console, which shows what Google last fetched.
Troubleshooting
- My pages dropped out of search: check for Disallow: / and test your main URLs.
- A blocked page still appears in results: robots.txt does not remove URLs. Allow crawling and add noindex.
- Google shows an old version: Google caches the file and refreshes it regularly. Request a refresh in Search Console.
- The file returns an error: serve it as plain text at the root of the domain with a 200 status.
- Rules seem ignored: check the crawler name, the case of the paths and whether a more specific rule overrides yours.
Privacy and how it works
Your browser builds the file from the options you choose and tests paths against it with your own copy of the matching rules. Nothing is uploaded, stored or logged. Print / save summary creates a short report from your entries, and the rest of the page is not included in the print.
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file at the root of your website that tells crawlers which parts of the site they may or may not visit. Search engines read it before they crawl your pages.
How do I create a robots.txt file for free?
Choose Allow everything, Block everything or Custom rules, add any paths to block, add your sitemap URL and copy the result. Save it as robots.txt and upload it to the root of your site, so it opens at yoursite.com/robots.txt.
Where do I put the robots.txt file?
At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file. A file in a subfolder is ignored by crawlers.
What does User-agent: * mean?
The user-agent line names the crawler a group of rules applies to. The asterisk means all crawlers that do not have their own group.
What is the difference between Disallow and Allow?
Disallow asks a crawler not to fetch a path. Allow makes an exception inside a disallowed folder. When rules overlap, Google uses the most specific rule, which is the longest matching path.
Does robots.txt remove a page from Google?
No. It stops crawling, but a blocked URL can still appear in results if other sites link to it. To keep a page out of search, allow crawling and use a noindex tag, or protect it with a password.
Should I block my admin area in robots.txt?
You can, but robots.txt is public, so it also tells people where your admin area is. Real protection needs a login and server settings. For WordPress the usual rule blocks /wp-admin/ and allows admin-ajax.php.
How do I block AI crawlers like GPTBot?
Add a group for each crawler with Disallow: / under its name. This tool has a list of common AI crawlers you can tick. It is a request, and crawlers that ignore the rules cannot be stopped this way.
Will blocking Google-Extended hurt my search ranking?
No. Google-Extended controls whether your content is used for some Google AI products. It does not affect Google Search crawling or ranking.
What is Crawl-delay?
Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and some others follow it. Google ignores it.
Should I list my sitemap in robots.txt?
Yes. A Sitemap line with the full URL helps crawlers find your sitemap. You can list more than one sitemap.
What do the * and $ characters do?
The asterisk matches any run of characters, and the dollar sign marks the end of a URL. For example, Disallow: /*.pdf$ blocks every URL that ends in .pdf.
Can I test my robots.txt here?
Yes. Enter paths or URLs, pick a crawler, and the tester shows whether each one is allowed or blocked, and which rule decided it. You can also paste another robots.txt to test.
Should I block CSS and JavaScript?
No. Google needs your CSS and JavaScript files to render your pages the way visitors see them. Blocking them can hurt how your pages are understood.
Can one wrong line break my site?
Yes. A line like Disallow: / under User-agent: * blocks every crawler from your whole site. The tool warns you when it sees that, and the tester helps you check important URLs.
Is my robots.txt uploaded anywhere?
No. The generator and tester run in your browser. Nothing you type is sent to Utiliby or to any other server.
What can I print or save from the page?
Print / save summary gives a short report with the settings, checks, test results and the file. Download file saves robots.txt. Copy code puts it on your clipboard.
Sources and further reading
- Google Search Central, Robots.txt introduction and specification (developers.google.com), for syntax and matching rules.
- RFC 9309, Robots Exclusion Protocol (rfc-editor.org), for the standard behind robots.txt.
- Google Search Central, Google crawlers overview, for crawler names including Google-Extended.
- Bing Webmaster Guidelines, Robots.txt and crawl control, for Crawl-delay behavior.
- OpenAI, Anthropic and Common Crawl crawler documentation, for GPTBot, ClaudeBot and CCBot.
- sitemaps.org Protocol, for the Sitemap line.
Sources were consulted in October 2026. Standards and rules can change, so always check the official page for your situation.