UtilibyWritten by the Utiliby editorial team · Last updated October 14, 2026 · Rules and formulas reviewed October 2026
What this free meta robots tag generator does
Not every page on a site should appear in search. Thank-you pages, internal search results, test pages and private areas usually do not belong there. The meta robots tag is the standard way to tell search engines what to do with a single page: show it or not, follow its links or not, and how much of it to display.
This free online meta robots tag generator lets you choose those rules with simple menus and tick boxes. It writes the tag for the head of your page, the matching X-Robots-Tag header, or ready-made Apache and Nginx rules for file types such as PDFs. A table explains every directive in plain English, and the checks point out conflicts before they cause trouble. It all runs in your browser, so nothing is uploaded.
How to use the generator
- Pick who the rule is for. All search engines is the usual choice. You can also address Google, Google News or Bing alone.
- Choose index and follow. Use noindex to keep the page out of results.
- Add options. Limit snippets, allow large images, or set an expiry date.
- Choose the output. A meta tag for pages, a header for any file, or a server rule for file types.
- Read the checks, then copy. Paste the tag in the head of the page or add the rule to your server.
You can start from a preset such as Thank-you page, Internal search results or PDF files, then adjust it.
Robots directives explained
| Directive | Effect |
|---|---|
| index / noindex | Allow or block the page in results |
| follow / nofollow | Allow or block following the page's links |
| none | Same as noindex, nofollow |
| noarchive | No saved copy link |
| nosnippet | No text or video snippet |
| max-snippet | Longest text snippet in characters |
| max-image-preview | none, standard or large image previews |
| max-video-preview | Longest video preview in seconds |
| noimageindex | Do not index images on the page |
| notranslate | No translation offer |
| unavailable_after | Stop showing after a date |
| indexifembedded | Allow indexing when embedded, with noindex |
Several of these are read by Google. Other search engines may support fewer, so treat the extras as requests, not guarantees.
Meta tag, header or robots.txt?
Use the meta robots tag for HTML pages when you can edit the head. Use the X-Robots-Tag header for files that have no head, such as PDFs, or when you want to apply a rule to many files from the server. Use robots.txt to control crawling of whole sections, for example to save crawl effort, but not to keep pages out of results.
The common trap is combining them. If robots.txt blocks a page, the crawler never loads it, so it never sees the noindex tag. The address can then stay in results with no description. To remove a page, let it be crawled and add noindex. When it has dropped out, you may block it if you wish.
Which pages should be noindex
- Thank-you and confirmation pages, which have no value in search.
- Internal search results, which can create endless thin pages.
- Staging and test sites, though a password is safer.
- Account, cart and checkout pages.
- Thin tag or date archives, kept as noindex, follow so links still count.
- Duplicate filtered views, although a canonical tag is often a better fix.
- Expired offers, using unavailable_after.
Do not noindex pages that you want to rank, and review the list from time to time, because a stray noindex on a template can remove an entire section from search.
Controlling how your page looks in results
The snippet options let you decide how much of your page is shown. Setting max-snippet to -1, max-image-preview to large and max-video-preview to -1 allows the fullest results, which often gets more attention. Setting nosnippet or a small max-snippet limits what is shown, which may suit paid content but can reduce clicks. Pick the setting that matches your business, not what looks neat. Changes apply after the page is crawled again.
Common mistakes
- A noindex tag left on after launch, often copied from a staging site.
- Blocking the page in robots.txt as well.
- Noindex on a page that is in your sitemap. Keep sitemaps for pages you want indexed.
- Using nofollow to hide pages. It does not remove them from results.
- Different tags for different bots that conflict. Google uses the most restrictive rule that applies to it.
- Using the tag outside the head.
Checking that it works
View the source of the live page and confirm the tag is in the head. For headers, use your browser's network panel or a header checker to confirm the X-Robots-Tag is sent for the file. In Google Search Console, the URL Inspection tool shows whether a page is indexable and whether noindex was detected. Allow time for a recrawl before expecting results to change.
Ready-made setups
These combinations cover most real situations. Press the matching preset to fill them in.
- Thank-you page: noindex, follow. It stays out of results and still passes visitors on.
- Staging site: noindex, nofollow, noarchive. A password or IP limit is the stronger protection, and the tag is a second line of defense.
- Internal search results: noindex, follow, so endless search pages do not fill the index.
- PDF price lists that should stay private: an X-Robots-Tag noindex rule for pdf files.
- Limited-time offer: unavailable_after set to the last day, so the page leaves results on time.
- News or magazine article: max-snippet -1, max-image-preview large and max-video-preview -1 for the fullest display.
- Members-only page: noindex, nofollow, noarchive, nosnippet and a login wall.
After you add a rule, request a recrawl in Search Console for important pages and keep a note of every noindex you add, so it is easy to audit later. A short list of noindex pages is far easier to maintain than a scattered one.
What robots directives cannot do
A robots directive is a polite request that major search engines follow, and it has limits. It does not stop people from visiting a page, and it does not stop a bad crawler from fetching it. Anything that must stay private needs a login, a password or a server rule that denies access. A noindex tag is also not instant: the page must be crawled again before it leaves results, and pages that are linked from many places can take longer to drop.
Directives also behave differently between engines. Google reads the full set described here, while other engines support fewer, so treat the extras such as max-image-preview and unavailable_after as Google features. When in doubt, check the documentation of each search engine you care about. Finally, remember that a rule on one page does not apply to others, so templates, plugins and server rules should be reviewed together. A site-wide plugin setting can override or duplicate a tag you added by hand, and conflicting tags are resolved in favor of the most restrictive one.
Privacy and how it works
The tag and server rules are built in your browser from your choices. The tool never visits your site, so it cannot tell whether the rule is already live. Nothing is uploaded, stored or logged. Print / save summary prints the report only.
Frequently asked questions
What is a meta robots tag?
It is a line of HTML in the head of a page that tells search engines whether to show the page in results and whether to follow its links.
How do I create a meta robots tag for free?
Choose index or noindex, follow or nofollow, add any extra options, and copy the tag. Paste it in the head of the page.
What does noindex do?
It asks search engines not to show the page in results. The page can still be visited by anyone who has the address.
What does nofollow do?
It asks search engines not to follow the links on the page or pass credit through them. For one link, use rel=nofollow on that link instead.
Is index, follow needed?
No. Those are the defaults, so a page with no robots tag is treated as index, follow. Writing them is harmless and makes the rule explicit.
Should I block a noindex page in robots.txt?
No. If a crawler cannot fetch the page, it never sees the noindex tag, and the address may still be listed. Allow crawling and use noindex.
What is the difference between robots.txt and noindex?
Robots.txt controls crawling. Noindex controls whether a page appears in results. To keep a page out of search, use noindex or a password.
What is X-Robots-Tag?
It is an HTTP header that carries the same directives as the meta tag. It works for files such as PDFs, images and videos that have no HTML head.
How do I noindex PDF files?
Pick the Apache or Nginx output, enter pdf as the file type and set noindex. Add the rule to your server configuration.
What do max-snippet and max-image-preview do?
They limit how much text, image or video a search engine may show in a result. They can also allow large previews, which can help click rates.
What is unavailable_after?
It tells Google to stop showing a page after a date and time, which suits events and limited offers.
What does nosnippet do?
It stops search engines from showing a text or video snippet for the page. It may also stop the page from being used in some features.
What does noarchive do?
It asks search engines not to offer a saved copy of the page. Google no longer shows a cache link, but other engines may.
How long does it take for noindex to work?
The search engine must crawl the page again before it sees the change. This can take days or weeks. Search Console can speed up a recheck for single pages.
Which pages should be noindex?
Common examples are thank-you pages, internal search results, staging sites, login and account pages, thin tag archives and filtered duplicates.
Is my data uploaded?
No. The tag is built in your browser and nothing is sent anywhere.
What does Print / save summary include?
A short report with who the rule applies to, the directives, the checks and the final code. The rest of the page is left out.
Sources and further reading
- Google Search Central, Robots meta tag, data-nosnippet and X-Robots-Tag specifications (developers.google.com).
- Google Search Central, Block search indexing with noindex.
- Bing Webmaster Guidelines, Robots meta tags supported by Bing.
- Apache HTTP Server documentation, Header directive and FilesMatch.
- Nginx documentation, add_header directive and location matching.
Sources were consulted in October 2026. Standards and rules can change, so always check the official page for your situation.