The robots meta tag gives you granular, per-page control over search engine indexing and link-following behavior — a more precise complement to the site-wide rules in robots.txt.
1The Robots Meta Tag Syntax
The robots meta tag lives in the <head> of an HTML document: <meta name="robots" content="...">. The content attribute accepts a comma-separated list of directives, the two most common being noindex (exclude from search results) and nofollow (don't follow or pass credit through this page's links).
By default, in the absence of this tag, every page is treated as index, follow — search engines assume they should index the page and follow its links unless told otherwise. The tag exists purely to override that default for specific pages.
2When To Use noindex
Typical noindex candidates include internal search result pages (which create infinite, low-value URL combinations), thank-you/confirmation pages, admin or staging environments accidentally left crawlable, and duplicate content variants that exist for UX reasons but shouldn't compete for rankings.
A critical operational pitfall: teams sometimes noindex their staging environment correctly, then forget to remove the tag when promoting to production, silently deindexing the live site. This is one of the most common, most damaging SEO incidents in real engineering organizations.
3X-Robots-Tag For Non-HTML Content
PDFs, images, and other non-HTML resources have no <head> to place a meta tag in, yet may still need indexing control — a downloadable internal PDF, for instance. The X-Robots-Tag HTTP response header provides the exact same directive vocabulary at the server level, applicable to any content type.
Many server frameworks and CDNs let you set this header conditionally by file path or content type, making it possible to noindex an entire /downloads/internal/ directory of PDFs without touching a single file.
4Step-by-Step Breakdown
Per-Page Instructions For Search Crawlers. The robots meta tag gives search engines explicit, per-page instructions about indexing and link-following — the difference between a thank-you page silently appearing in search results and staying exactly where it belongs, out of them.
noindex Keeps A Page Out Of Search Results. Adding <meta name="robots" content="noindex"> to a page tells every compliant crawler not to include it in search results, even if other pages link to it. It's the standard tool for thank-you pages, internal search results, and staging environments.
noindex Use Case. A checkout 'Thank you for your order' page shouldn't appear in Google search results. What's the correct fix?
- →<meta name="robots" content="noindex">
- →<meta name="robots" content="nofollow"> alone
- →There's no way to prevent this; delete the page
nofollow Controls Link Equity Flow. content="nofollow" tells crawlers not to follow (or pass ranking credit through) any links on that page. It's distinct from noindex — a page can remain fully indexed while still telling crawlers not to traverse its outbound links.
Combining Directives. Can noindex and nofollow be used together in a single robots meta tag?
- →Yes, as a comma-separated list of independent directives
- →No, they are mutually exclusive and cannot combine
- →Only nofollow is valid in a meta tag; noindex requires a header
X-Robots-Tag: The HTTP Header Equivalent. Non-HTML files (PDFs, images) can't carry a <meta> tag, so the same directives are available as an HTTP response header, X-Robots-Tag, set at the server level — functionally identical to the meta tag but applicable to any content type.
Non-HTML Robots Control. A site wants to prevent a downloadable PDF from being indexed. Since PDFs can't contain a <meta> tag, what's the correct mechanism?
- →robots.txt disallow rules are the only option
- →The X-Robots-Tag HTTP response header
- →It's impossible to control indexing of non-HTML files
Crawler Control Mastered. You can now control exactly which pages appear in search results and which outbound links pass ranking credit, using both the HTML meta tag and its HTTP-header equivalent for non-HTML resources.
Control Search Indexing With Robots Meta. The robots meta tag tells search engines whether to index this page and follow its links.
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1Robots Directives Don't Affect Assistive Technology Access
noindex and nofollow only instruct search engine crawlers; the page remains fully accessible to screen reader users and other visitors who reach it directly, so it's not a substitute for actual access control.
SEO Implications
- 1
noindex Is The Correct Tool For Duplicate Or Low-Value Pages, Not robots.txt Disallow
Blocking a page in robots.txt prevents crawling but doesn't guarantee removal from an index if it's already indexed or heavily linked; noindex is the directive that reliably controls index inclusion.
- 2
An Accidental noindex On Production Is One Of The Most Damaging, Most Common SEO Incidents
Because the tag is invisible in the rendered page and easy to leave in place after an environment promotion, teams should include a check for it in deployment or CI pipelines.
Best Practices
Automate A Check For noindex Tags In Production Deploy Pipelines
It catches the single most common and costly robots-meta mistake — a staging-only noindex tag accidentally shipping to the live production domain — before it causes real ranking damage.
Use X-Robots-Tag, Not robots.txt, To Control Indexing Of Non-HTML Files
robots.txt only prevents crawling, not indexing of already-known URLs; X-Robots-Tag provides the same reliable index-control guarantee that meta robots gives HTML pages.
Frequent Bugs
A site's entire production domain silently disappears from Google search results after a deploy.
A noindex tag meant only for staging shipped to production. Add an automated pre-deploy check that fails the build if noindex is detected on the production environment.
An internal PDF report keeps appearing in search results despite requests to remove it.
PDFs can't carry a meta tag. Set the X-Robots-Tag: noindex HTTP header for that file path at the server or CDN level instead.
Real-World Examples
Environment-Conditional Robots Meta
A Next.js app that only renders noindex outside of production, preventing the most common accidental-deindexing incident.
<meta
name="robots"
content={process.env.NODE_ENV === 'production' ? 'index, follow' : 'noindex, nofollow'}
/>