Learn
Indexability Debugging Guide: Why Google Cannot Index Your Page
A practical workflow for pages that are crawled but not indexed: what the Search Console states mean, the signals that decide indexing, and how to fix each cause.
Run a fresh DomainLens audit and use the report as your priority list.
Crawled but not indexed: what the states mean
Crawling and indexing are two different steps. Google can fetch a URL (crawl it) and still decide not to store it in the index, which is why a page can return 200, appear fine in a browser, and never show up in search. The Page Indexing report in Search Console names the exact state, and the fix depends entirely on which one you see.
The common states each mean something specific: "Crawled – currently not indexed" usually signals a quality or duplication judgement; "Discovered – currently not indexed" points to crawl budget or server load; "Excluded by ‘noindex’ tag" is a directive you set; "Alternate page with proper canonical tag" means Google chose a different URL as canonical; and "Duplicate, Google chose different canonical" means your canonical hint was overridden.
The signals that control indexing
A page is indexable only when every one of these agrees it should be. A single conflicting signal is enough to keep it out.
- HTTP status: the URL must return 200. A 3xx, 4xx, or soft 404 removes it from consideration.
- robots.txt: it must not disallow the path — note that a blocked page can still be indexed URL-only but cannot have its noindex read.
- Robots meta / X-Robots-Tag: neither the HTML meta robots nor the HTTP header may say noindex.
- Canonical: the page’s canonical should point to itself, not to another URL that then absorbs it.
- Content quality and uniqueness: thin, boilerplate, or near-duplicate pages get crawled and dropped.
How to debug one URL, step by step
- Run the URL through Search Console’s URL Inspection and read the exact status and the "Coverage" reason.
- Use "Test live URL" and view the rendered HTML — confirm the content and the robots meta actually present to Googlebot, not just to your browser.
- Fetch the response headers (curl -I) and check for an X-Robots-Tag: noindex that no HTML view would reveal.
- Confirm robots.txt allows the path, then confirm the canonical in the rendered HTML points to this URL.
- If everything passes and it is still "Crawled – not indexed", judge the content against the query intent: is it genuinely worth a slot, or a near-duplicate of a template?
How to fix each cause
- noindex you did not intend: remove the meta tag or the X-Robots-Tag header, then request indexing.
- Wrong canonical: point the canonical at the page itself and make internal links use the same canonical URL.
- Blocked in robots.txt but you want it indexed: unblock the path — do not use robots.txt to deindex, use noindex instead.
- Duplicate/near-duplicate: consolidate with a canonical to the primary version, or make the content genuinely distinct.
- Discovered – not indexed: reduce crawl waste (fewer low-value URLs, faster server) so Google reaches the page, and strengthen internal links to it.
Mistakes that keep pages out of the index
- Using robots.txt to remove a page — a disallowed page keeps any existing index entry because Google can no longer read the noindex.
- Auditing source HTML only, when the noindex or canonical is injected by JavaScript after render.
- Requesting indexing repeatedly instead of fixing the signal that is blocking it.
- Assuming a 200 status means indexable, ignoring a soft 404 where the page says "not found" in the body.
How to confirm the page is indexable
After the fix, use "Test live URL" again and confirm the rendered HTML, the headers, and the canonical all agree the page should be indexed, then request indexing and watch the state change in the Page Indexing report over the following days.
DomainLens surfaces these signals in one place — status code, robots directives, canonical target, and whether they conflict — so you can spot the contradiction before you wait days for Search Console to confirm what is wrong.