Meta robots checker

Check whether the page is blocked from indexing via meta robots tags or X-Robots-Tag header

Free
No sign-up
Instant results

Check results

This check only covers indexing directives. For a full picture of your page, run a page audit.

For issues across your whole site — duplicate titles, orphan pages, broken internal links — run a site audit.

Want us to fix what we found? Our team can help.

What is the meta robots tag and why it matters

The <meta name="robots"> tag and the X-Robots-Tag HTTP header tell search engines how to handle a page: whether to index it, whether to follow its links, whether to show a snippet or image preview in results. A single incorrect directive — accidentally left after staging, or added by a plugin — can remove an entire page from Google's index. This is one of the most common and most damaging SEO mistakes because it's invisible to the naked eye: the page loads normally for users, but search engines quietly drop it.

What this tool checks

  • Meta robots tag — indexing, follow, snippet, preview, and expiration directives
  • Meta googlebot tag — Google-specific overrides
  • X-Robots-Tag HTTP header — server-side equivalent, often used for non-HTML files (PDFs, images)
  • Multiple tags — whether the page has more than one conflicting meta robots tag
  • Conflicts — mismatches between the generic meta robots tag and the bot-specific meta googlebot tag
  • Unknown directives — typos that search engines silently ignore

Core directives

  • noindex — page is excluded from the index
  • nofollow — links on the page don't pass equity and aren't followed
  • none — shorthand for noindex, nofollow
  • all — shorthand for index, follow (default, rarely written explicitly)
  • noarchive — prevents a cached copy. Note: Google removed the Cache feature in 2024, so this directive has minimal practical effect today

SERP-appearance directives (Google, 2019+)

  • nosnippet — no text snippet shown in search results. Reduces CTR significantly
  • max-snippet:N — limits snippet length to N characters. max-snippet:0 effectively disables the snippet
  • max-image-preview:none | standard | large — controls the image thumbnail shown next to the result
  • max-video-preview:N — limits video preview length in seconds. max-video-preview:0 disables it
  • noimageindex — images on this page are not indexed in Google Images
  • notranslate — Google doesn't offer translation in SERP
  • unavailable_after:DATE — page is automatically removed from the index after a specific date
  • indexifembedded — allows indexing when the page is embedded in an iframe (even with noindex on the parent)

Good vs bad examples

Good — default, no restrictions (this tag is usually omitted entirely):

<meta name="robots" content="index, follow">

Good — a staging/login page intentionally hidden:

<meta name="robots" content="noindex, nofollow">

Bad — a production landing page with noindex left over from development:

<meta name="robots" content="noindex">

Bad — a typo that search engines silently ignore:

<meta name="robots" content="no-index">

Common mistakes

  • noindex left after staging — most common cause of "my new site isn't in Google"
  • CMS or SEO plugin added noindex automatically — often happens on category/archive pages after plugin updates
  • X-Robots-Tag on the server overrides meta tags — a header set in nginx/htaccess silently overrides whatever the HTML says
  • Conflicting meta robots and meta googlebot tags — the bot-specific tag wins for its bot; unintentional divergence leads to surprises
  • Typos in directives — no-index, noindeex, no index are all silently ignored. Always verify by crawling the URL in Google Search Console

Frequently asked questions

robots.txt controls crawler access to the entire site or its sections. The meta robots tag controls indexing of a specific page. A page can be accessible to crawlers (not blocked in robots.txt) but still excluded from indexing via noindex.
If both are specified and conflict, the search engine typically applies the stricter rule. For example, if the meta tag allows indexing but X-Robots-Tag forbids it, the page won't be indexed.
After adding noindex, the page is excluded from search results on the next crawl — usually a few days to a few weeks depending on how often your site is crawled. To force faster removal, submit the URL in Google Search Console or use the "Removals" tool.
Several possible causes: an X-Robots-Tag HTTP header blocking indexing at the server level, a Disallow rule in robots.txt preventing the crawler from reaching the page, a canonical tag pointing to a different URL, thin or duplicate content, or the page simply being too new and not yet crawled. Check all of these in Google Search Console's URL Inspection tool.
For Google, the bot-specific tag (meta name="googlebot") takes priority over the generic meta name="robots". So if meta robots says noindex but meta googlebot says index, Google will index the page. Other crawlers that don't recognize the googlebot-specific tag will follow meta robots.
Yes — that's actually its primary use case. Meta tags can only be placed inside HTML documents, so to control indexing of PDFs, images, videos, and other binary files, you must use X-Robots-Tag. Typically configured in the web server (nginx add_header, Apache .htaccess Header set) based on file extension.

Related checks

Other indexing checks you might want to run.

robots.txt

Verify that robots.txt is accessible and doesn't block important pages.

sitemap.xml

Check sitemap availability, format, and URL count.

Canonical

Check the canonical tag and whether it points to the correct URL.

HTTP → HTTPS

Verify that HTTP traffic is redirected to HTTPS.

www redirect

Check whether www and non-www versions redirect to one canonical host.

Server 404

Check that the server returns a proper 404 for non-existent URLs.
Browse all 28 checkers →