Lernen
Robots.txt File: What It Controls and a Safe Example
robots.txt is a crawl-access file. It can reduce unnecessary fetching, but it is not a privacy, deindexing, or canonicalization control.
Starte ein frisches DomainLens-Audit und nutze den Report als Prioritätenliste.
What the robots.txt file does
A robots.txt file lives at the root of a host, for example https://example.com/robots.txt. It gives compliant crawlers a set of paths they may or may not fetch. It is useful for preventing waste on predictable URL spaces such as internal search, but it does not protect a private page and it cannot force Google to remove an already known URL.
The most expensive mistake is using Disallow together with a noindex tag. When Google cannot fetch the page, it may not see the noindex. Choose crawl control or index removal according to the result you need.
User-agent: *
Disallow: /search/
Disallow: /account/
Sitemap: https://example.com/sitemap.xmlTest rules against the final URL
Validate the live file with the robots.txt checker , decide removal behavior with the noindex versus robots.txt guide , and confirm the affected page with the indexability checker .
- Keep CSS and JavaScript needed to render public pages crawlable.
- Use one sitemap declaration with the final canonical sitemap URL.
- Check both www and non-www hosts if both resolve.