Learn
Indexability Debugging Guide: Why Google Cannot Index Your Page
A practical workflow for pages that are crawled but not indexed: what the Search Console states mean, the signals that decide indexing, and how to fix each cause.
Run a fresh DomainLens audit and use the report as your priority list.
Crawled but not indexed: what the states mean
Crawling and indexing are two different steps. Google can fetch a URL (crawl it) and still decide not to store it in the index, which is why a page can return 200, appear fine in a browser, and never show up in search. The Page Indexing report in Search Console names the exact state, and the fix depends entirely on which one you see.
The common states each mean something specific: "Crawled – currently not indexed" usually signals a quality or duplication judgement; "Discovered – currently not indexed" points to crawl budget or server load; "Excluded by ‘noindex’ tag" is a directive you set; "Alternate page with proper canonical tag" means Google chose a different URL as canonical; and "Duplicate, Google chose different canonical" means your canonical hint was overridden.
The signals that control indexing
A page is indexable only when every one of these agrees it should be. A single conflicting signal is enough to keep it out.
- HTTP status: the URL must return 200. A 3xx, 4xx, or soft 404 removes it from consideration.
- robots.txt: it must not disallow the path — note that a blocked page can still be indexed URL-only but cannot have its noindex read.
- Robots meta / X-Robots-Tag: neither the HTML meta robots nor the HTTP header may say noindex.
- Canonical: the page’s canonical should point to itself, not to another URL that then absorbs it.
- Content quality and uniqueness: thin, boilerplate, or near-duplicate pages get crawled and dropped.
How to debug one URL, step by step
- Run the URL through Search Console’s URL Inspection and read the exact status and the "Coverage" reason.
- Use "Test live URL" and view the rendered HTML — confirm the content and the robots meta actually present to Googlebot, not just to your browser.
- Fetch the response headers (curl -I) and check for an X-Robots-Tag: noindex that no HTML view would reveal.
- Confirm robots.txt allows the path, then confirm the canonical in the rendered HTML points to this URL.
- If everything passes and it is still "Crawled – not indexed", judge the content against the query intent: is it genuinely worth a slot, or a near-duplicate of a template?
How to fix each cause
- noindex you did not intend: remove the meta tag or the X-Robots-Tag header, then request indexing.
- Wrong canonical: point the canonical at the page itself and make internal links use the same canonical URL.
- Blocked in robots.txt but you want it indexed: unblock the path — do not use robots.txt to deindex, use noindex instead.
- Duplicate/near-duplicate: consolidate with a canonical to the primary version, or make the content genuinely distinct.
- Discovered – not indexed: reduce crawl waste (fewer low-value URLs, faster server) so Google reaches the page, and strengthen internal links to it.
Mistakes that keep pages out of the index
- Using robots.txt to remove a page — a disallowed page keeps any existing index entry because Google can no longer read the noindex.
- Auditing source HTML only, when the noindex or canonical is injected by JavaScript after render.
- Requesting indexing repeatedly instead of fixing the signal that is blocking it.
- Assuming a 200 status means indexable, ignoring a soft 404 where the page says "not found" in the body.
How to confirm the page is indexable
After the fix, use "Test live URL" again and confirm the rendered HTML, the headers, and the canonical all agree the page should be indexed, then request indexing and watch the state change in the Page Indexing report over the following days.
DomainLens surfaces these signals in one place — status code, robots directives, canonical target, and whether they conflict — so you can spot the contradiction before you wait days for Search Console to confirm what is wrong.
When the problem is crawl budget, not the page
Everything above assumes Google reached the URL and made a decision about it. A large share of unindexed pages never got that far: Search Console reports them as "URL is unknown to Google", which means no crawl happened, so no indexing decision exists to debug.
That is a budget problem, and it has a different shape. Google allocates a finite number of requests per site per day, split across HTML documents, JavaScript, CSS, and images. If the pages advertise many subresources, most of that allowance is consumed fetching assets rather than discovering documents — a site can be crawled thousands of times a quarter and still leave hundreds of URLs unvisited.
| Symptom | What it means | Where to look |
|---|---|---|
| URL is unknown to Google | Never crawled; no decision made | Crawl stats, sitemap submission |
| Crawled — currently not indexed | Crawled and declined | Page quality, duplication, site authority |
| Discovered — currently not indexed | Queued but deprioritised | Crawl budget, internal linking |
Faceted navigation: the usual source of the waste
Filter and sort parameters multiply URLs combinatorially. Four filters with five values each produce over six hundred combinations of one category page, nearly all of them near-identical, and every one of them is a URL Google may spend a request on.
The goal is not to block crawling outright — a blocked URL keeps whatever index status it already had, because Google cannot read the noindex on a page it may not fetch. The goal is to make a small, deliberate set of filter pages crawlable and to keep the rest out of the crawlable link graph in the first place.
- 1Decide which filter combinations have real search demand — usually a handful, such as one brand or one size, never a combination of four.
- 2Give those a clean, static URL and link them from the category page as ordinary anchors.
- 3Render every other combination behind a control Googlebot does not follow, so the URL is never discovered.
- 4Set a self-referencing canonical on the deliberate landing pages, and canonicalise accidental parameter variants to the unfiltered category.
- 5Confirm in Crawl stats that request volume against the parameter space falls over the following weeks.
Orphan pages: reachable by sitemap, invisible to the link graph
A page listed in a sitemap but linked from nowhere is an orphan. The sitemap gets it discovered; it does not get it valued. Internal links are how a site says which of its pages matter, and a page with no incoming links is making no claim at all.
Orphans are common after a migration, a CMS change, or a batch of pages published straight from a template. They usually show up as a cluster of URLs that stay "Discovered" or "Crawled — currently not indexed" indefinitely while the rest of the section indexes normally.
- Crawl the site with a tool that follows links only, then compare the result against the sitemap — anything in the sitemap but not the crawl is an orphan.
- Link each orphan from the section index and from at least one closely related article, using anchor text that describes the destination.
- If nothing on the site wants to link to a page, that is evidence the page should be merged into one that does, rather than propped up.
Link discovery and index bloat: inspect the graph before requesting another crawl
Google discovers most URLs through ordinary anchors. A control that looks like a link but is a button, a div with a click handler, or an href containing javascript:void(0) gives a crawler no destination to fetch. This is why a page can appear in a menu to users yet remain unknown to Google: the navigation has interaction, but not a link graph.
The reverse problem is index bloat. Filter combinations, internal search pages, tracking parameters and template variants can create thousands of crawlable URLs that no one needs in search. Do not solve that by blocking every URL in robots.txt: Google then cannot read a noindex directive. Keep the small set of valuable pages in ordinary links and use noindex, canonicalisation or removal for the rest according to what the URL represents.
Use the broken link checker to verify that anchors resolve, the indexability checker to confirm the directive and canonical on a sample URL, and the internal linking strategy guide when deciding where a strong contextual link belongs.
| What you find | Evidence to collect | Fix |
|---|---|---|
| A clickable element without href | View source or rendered HTML and compare it with the navigation a user can click | Render an anchor with a resolvable URL; JavaScript can still intercept the click |
| A sitemap URL with no incoming internal link | Compare a link crawl with the sitemap | Add a contextual link from its hub and a related article, or merge the page |
| Many near-identical parameter URLs | Group Search Console examples by URL pattern and inspect their canonical/noindex response | Keep only demand-backed landing pages crawlable; consolidate or noindex the rest |