SEO tools
Robots.txt Checker: Test What Crawlers Are Allowed to Fetch
Fetches the robots.txt file for the host you enter, reports whether it exists and whether it blocks every crawler, and reads the page-level robots directives alongside it.
What this robots.txt checker tests
The three are checked together on purpose. Crawl control and index control are separate mechanisms that people routinely mistake for each other, and a directive in one place is frequently contradicted by a directive in another.
- Whether /robots.txt exists on the host and returns 200 rather than an HTML error page.
- Whether the wildcard user-agent group blocks the entire site with Disallow: /.
- The page's meta robots directive, which controls indexing rather than crawling.
- The X-Robots-Tag response header, which does the same job for any resource type.
Reading the result
A file that returns 500 is worse than one that is absent: Google may pause crawling the whole host until it recovers, whereas a clean 404 is treated as "nothing is blocked".
| Finding | What it means | Usual fix |
|---|---|---|
| robots.txt missing | No file, or it does not return 200 | Add one, if only to declare the sitemap |
| Blocks all crawlers | Disallow: / in the wildcard group | Remove it — staging rules often ship by accident |
| meta robots noindex | The page asks to be kept out of the index | Confirm whether that is intentional |
| X-Robots-Tag noindex | The header does the same at server level | Check the server or CDN config, not the HTML |
What a robots.txt file cannot do
robots.txt controls crawl access and nothing else. It cannot remove a page from search, cannot keep anything private, and cannot consolidate duplicates. Blocking a URL actively prevents Google from seeing a noindex tag on it, which is why the two together leave pages stuck in the index. See noindex vs robots.txt for choosing the right control, and robots.txt examples for the syntax.
- Does this checker test a specific user agent?
- It reads the wildcard group, which is what applies to any crawler without rules of its own. A bot with a dedicated group obeys only that group, so check bot-specific rules by reading the file directly.
- Why does it say the file is missing when I can open it?
- Most often the URL returns a 200 HTML page rather than plain text, or the file exists on www while the check followed the non-www host. Both count as no usable robots.txt.
Other tools
Looks for the sitemap the way a crawler does — the robots.txt declaration first, then the conventional locations — then confirms it parses as XML and counts the URLs inside it.
Reads the canonical URL a page declares, then fetches that URL to confirm it resolves, is not redirected, and is not noindexed — the three ways a canonical silently becomes inert.
Parses the structured data on the page you enter, lists the schema.org types it finds, and flags blocks that fail to parse or are missing the properties their type requires.
Follows the URL you enter through every hop, reporting the status code at each step and the final destination — the detail a browser hides once the address bar settles.
Fetches the page you enter, takes a sample of the internal links it points at, and requests each one — reporting the status code that comes back instead of the colour of the link.
Checks every signal that decides whether a page is allowed into the index — the status code, robots.txt access, meta robots, the X-Robots-Tag header, and the canonical.
Queries the live DNS for the domain you enter and reports the records it publishes: the addresses it resolves to, where its mail goes, and whether the TXT records email authentication depends on are present.
Queries the registry over RDAP — the structured successor to classic WHOIS — and reports when the domain was registered, how old it is now, who the registrar is, and when the registration lapses.
Enter a domain to find when it was registered, how old it is today, when it expires, and which registrar holds it. Domain age is context for a diagnosis, not a Google ranking factor.
Opens a TLS connection to the host you enter and reports the certificate it presents — the issuer, the validity window, the days remaining, and the negotiated protocol version.
Requests the URL you enter and reports what the server sends back in its headers — the security headers present, the caching policy, whether the response is compressed, and which HTTP version was negotiated.