Learn
Robots.txt Example: Allow All, Disallow All, and Common Rules
A robots.txt file controls crawling, never indexing. Most damage comes from two lines that look almost identical and do opposite things.
Run a fresh DomainLens audit and use the report as your priority list.
Robots.txt file syntax
The file lives at the root of a host, as /robots.txt, and applies only to that host and scheme. It is a plain text file read top to bottom as groups: a User-agent line opens a group, and the Disallow, Allow, and Sitemap lines that follow belong to it.
Two rules explain most confusion. A crawler obeys the single most specific User-agent group that matches it and ignores every other group, so a rule in the wildcard group does not apply to a bot that has its own group. And within a group, the most specific matching path rule wins rather than the first one listed.
| Directive | Purpose | Notes |
|---|---|---|
| User-agent | Opens a group for one crawler | * matches any bot without its own group |
| Disallow | Requests a path is not fetched | Empty value means nothing is blocked |
| Allow | Carves an exception out of a Disallow | More specific path wins |
| Sitemap | Declares a sitemap URL | Absolute URL; belongs to no group |
| Crawl-delay | Requests slower crawling | Google ignores it |
Robots.txt example: allow all
Allowing everything can be written two ways, and both are correct. An empty Disallow value means "nothing is disallowed", which reads oddly but is the form the original specification defines.
- An absent robots.txt file has the same practical effect as allow-all, provided the URL returns 404.
- A robots.txt that returns 500 is not treated as allow-all — Google may stop crawling the site entirely until it recovers.
- Declare the sitemap even in an allow-all file; it is the only place a crawler is told where to find it.
# Explicit form — nothing is blocked
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlRobots.txt example: disallow all
Blocking everything differs from the allow-all form by a single character, which is why the two get confused with expensive consequences. A stray slash after Disallow blocks the whole site.
This form belongs on staging environments and nowhere else. It is not a way to keep pages out of search results — see the next section for why.
# Note the slash — this blocks everything
User-agent: *
Disallow: /
# The difference in one character:
# Disallow: -> nothing is blocked
# Disallow: / -> everything is blockedCommon robots.txt examples by use case
Real files are usually a handful of narrow rules rather than a broad block. The examples below cover the patterns that come up most often.
- The * wildcard matches any sequence of characters; $ anchors the end of the URL.
- Paths are case-sensitive, and matching is on the path only — the query string counts as part of it.
- Never block CSS or JavaScript that the page needs to render, or Google will index a broken version of it.
- Never list a private path here. robots.txt is public, so it advertises exactly what you wanted hidden.
# Block a directory but keep one subpath crawlable
User-agent: *
Disallow: /internal/
Allow: /internal/public-report/
# Block URLs with a tracking parameter
User-agent: *
Disallow: /*?utm_
# Block a file type
User-agent: *
Disallow: /*.pdf$
# Give one bot different rules from everyone else.
# Googlebot obeys ONLY its own group — the wildcard
# group below does not apply to it at all.
User-agent: Googlebot
Disallow: /drafts/
User-agent: *
Disallow: /drafts/
Disallow: /searchRobots.txt checker and tester: what to verify
Testing means checking real URLs against the file rather than reading it and assuming. The failure modes are quiet: a file that returns HTML, a rule matching more than intended, or a group that a bot never reads because it has its own.
- 1Confirm /robots.txt returns 200 with a text/plain content type — a soft 404 that returns an HTML page is treated as unparseable.
- 2Check that it lives on the exact host and scheme you care about; https://example.com and https://www.example.com read different files.
- 3Test specific URLs, including at least one you expect blocked and one you expect allowed, rather than only reading the rules.
- 4Verify every rendering resource — CSS, JS, fonts, images — is crawlable.
- 5Confirm the Sitemap line is an absolute URL that itself returns 200 and is not redirected.
# Status and content type
curl -sI https://example.com/robots.txt | head -n 3
# The file itself
curl -s https://example.com/robots.txt
# Just the sitemap declarations
curl -s https://example.com/robots.txt | grep -i '^sitemap'Robots.txt and SEO: what it cannot do
robots.txt manages crawl access and nothing else. It cannot deindex a page, cannot enforce privacy, and cannot consolidate duplicates. Choosing it for any of those jobs produces the opposite of the intended result — the comparison in noindex vs robots.txt covers which control fits which outcome, and robots.txt best practices covers the wider strategy. For the sitemap it declares, see the XML sitemap guide.
- What is the difference between "Disallow:" and "Disallow: /"?
- An empty value blocks nothing and is the standard allow-all form. A single slash blocks the entire site. One character separates the two, which is why staging rules so often reach production.
- Does robots.txt remove a page from Google?
- No. It requests that the page is not fetched. A blocked URL can still appear in results from external links, and blocking prevents Google from ever seeing a noindex tag on it.
- Where must the robots.txt file be located?
- At the root of the host, as /robots.txt. It applies only to that host and scheme, so subdomains and http versus https each need their own file.
- Do I need a robots.txt file at all?
- Not strictly — an absent file returning 404 means everything is crawlable. It is still worth having one to declare the sitemap, and a file returning 500 is actively harmful.