robots.txt is the first file well-behaved crawlers check on any domain, declaring which paths are open or closed to crawling — the site-wide counterpart to the per-page robots meta tag, with its own important limitations.
1Core Syntax: User-agent And Disallow
A robots.txt file consists of one or more rule blocks. Each block starts with User-agent:, naming which crawler the following rules apply to — a specific bot name, or * as a wildcard matching all crawlers. Disallow: lines then list path prefixes that crawler should not request.
An Allow: directive can carve out exceptions within a broader disallowed path, useful for blocking an entire directory except one specific file or subfolder within it. An empty Disallow: line (with nothing after the colon) means no restriction at all for that user-agent.
2Why robots.txt Is Not Security
A recurring, damaging misconception is treating robots.txt as a way to 'hide' sensitive paths. Two facts make this dangerous: first, robots.txt is a fully public file — anyone, including attackers, can simply request /robots.txt and read every disallowed path listed. Second, Disallow rules are entirely voluntary; compliant crawlers like Googlebot honor them, but nothing prevents a malicious scraper or attacker's tooling from ignoring the file completely and requesting the paths anyway.
In practice, listing a sensitive path in robots.txt can be actively counterproductive — it draws attention to exactly the paths an attacker would want to investigate first. Real access control requires actual authentication and authorization at the application or server level.
3Disallow Blocks Crawling, Not Necessarily Indexing
This is one of the most frequently misunderstood distinctions in technical SEO. Disallow tells crawlers not to *fetch* a URL's content — but if that URL is discovered another way (most commonly, an external site linking to it), a search engine can still choose to index it based on the link context alone, typically displaying it in results with no readable description, since the crawler was never permitted to actually read the page.
For a page that must reliably never appear in search results, noindex (from the earlier Robots Meta lesson) is the correct tool — but critically, a noindex tag only works if the crawler is allowed to actually reach and read the page. A page both Disallow-ed in robots.txt and noindex-tagged creates a contradiction: the crawler can never see the noindex instruction because it's blocked from fetching the page at all.
4Step-by-Step Breakdown
The First File Every Crawler Checks. Before crawling any page on a domain, well-behaved bots first request /robots.txt — a plain-text file at the site root declaring which paths they're permitted to crawl. It's the site-wide counterpart to the per-page robots meta tag covered earlier in this module.
User-agent And Disallow: The Core Syntax. A robots.txt rule block starts with User-agent (which crawler the rule applies to, or * for all), followed by one or more Disallow lines listing path prefixes that crawler shouldn't request. An empty Disallow means no restriction at all.
Robots.txt Core Syntax. What does 'Disallow: /admin/' instruct compliant crawlers to do?
- →Not to crawl any URL starting with /admin/
- →To crawl but never index anything under /admin/
- →To request deletion of the /admin/ path from the server
robots.txt Is Not A Security Mechanism. Disallowing a path in robots.txt is a polite request compliant crawlers honor — it does nothing to actually restrict access. Malicious bots ignore it entirely, and the file itself is public, meaning listing sensitive paths there can actually advertise their existence to attackers.
robots.txt Security Limitations. A team disallows /admin-panel/ in robots.txt to 'hide it from hackers.' What's wrong with this approach?
- →Nothing; this is a fully valid security measure
- →robots.txt is publicly readable and only voluntarily honored, so it can actually advertise the path to attackers
- →The syntax shown is simply invalid
Disallow Doesn't Guarantee No-Index. A page blocked by robots.txt can still appear in search results (typically with no description, since the crawler was never allowed to read it) if it's discovered through external links — Disallow prevents crawling, not necessarily indexing, which is why noindex remains the correct tool for guaranteed exclusion.
Disallow vs Indexing. A page is blocked via robots.txt Disallow but has many external backlinks. Can it still appear in search results?
- →No, Disallow guarantees it can never appear in results
- →Yes, it can still appear, typically with no description since it was never crawled
- →Only if the site owner explicitly requests indexing separately
robots.txt Fluency Achieved. You now understand the core User-agent/Disallow syntax, why robots.txt is not a security mechanism despite common misuse, and the crucial distinction between blocking crawling and guaranteeing exclusion from search results.
Block A Page From Indexing. A per-page robots meta tag can override site-wide robots.txt rules for this one page.
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1robots.txt Only Affects Search Crawlers, Never Real User Or Assistive Technology Access
Disallowing a path in robots.txt has zero effect on whether a human visitor, including one using a screen reader, can actually reach and use that page directly.
SEO Implications
- 1
robots.txt Is The Correct Tool For Conserving Crawl Budget, Not Guaranteeing Exclusion
It's ideal for preventing crawlers from wasting attention on infinite filter/sort URL combinations or internal search results, but noindex remains the correct tool when guaranteed search-result exclusion is the actual goal.
- 2
A Disallow And noindex Combination On The Same Page Is Self-Defeating
Since the crawler can never fetch the page to read its noindex tag if robots.txt blocks it first, this common misconfiguration can paradoxically leave a page indexed with no readable snippet instead of fully excluded.
Best Practices
Never Use robots.txt As A Substitute For Actual Access Control
It's a publicly readable, voluntarily-honored file — real security for sensitive paths requires authentication and authorization at the application layer, not a crawler-etiquette file.
Use robots.txt For Crawl Budget Conservation, noindex For Guaranteed Exclusion
Understanding this division of labor prevents the common, self-defeating mistake of combining Disallow and noindex on the same page, which prevents the noindex instruction from ever being read.
Frequent Bugs
A team disallows a directory in robots.txt hoping to keep it 'secret', then discovers it was scraped by an unrelated bot anyway.
robots.txt is publicly readable and only voluntarily honored. Implement actual authentication for any genuinely sensitive path.
A page shows up in search results with no title or description, despite being blocked in robots.txt.
This is expected behavior when an externally-linked, robots.txt-disallowed page gets indexed via link context alone. To fully exclude it, allow crawling and use noindex instead.
Real-World Examples
Standard Production robots.txt
A typical, correctly-scoped robots.txt for a content site, blocking low-value crawl paths while declaring the sitemap.
User-agent: *
Disallow: /search
Disallow: /*?sort=
Sitemap: https://example.com/sitemap.xml