Guías
Crawl Budget: When It Matters and When It Does Not
Crawl budget is real and almost certainly not your problem. Google says it below about a million URLs, and most sites discussing it have a few thousand.
Ejecuta una auditoría nueva en DomainLens y usa el informe como lista de prioridades.
What it is, and who it affects
Crawl budget is the number of URLs Googlebot will fetch from your site in a given period. It is determined by two things: how much crawling your server can take without slowing down, and how much Google wants to crawl based on demand and freshness.
Google has stated the threshold plainly: sites under a few thousand URLs are almost always crawled efficiently, and crawl budget becomes a real consideration for large sites — the guidance mentions the region of a million URLs, or large sites with frequently changing content.
When it genuinely applies
Note that three of those five are not about size. A small site with a slow server or heavy parameter generation can waste crawl in ways that matter.
- Large e-commerce catalogues, especially with faceted navigation generating combinatorial URLs.
- News and publishing sites where freshness matters and new URLs appear constantly.
- Sites with a large share of low-value URLs — parameters, internal search, session IDs.
- Slow servers, where Google reduces crawl rate to avoid causing problems.
- Sites with many redirect chains, each consuming fetches without yielding a page.
Finding out whether it affects you
The last step is the decisive one. Crawl budget is only a problem when it delays or prevents indexing — the statistics are interesting and the outcome is what matters. See log file analysis for reading what Googlebot actually fetched.
- 1Open the Crawl Stats report in Search Console and read total requests, average response time and the breakdown by response code.
- 2Compare crawl requests against the number of URLs you actually want indexed. Far more requests than URLs means waste.
- 3Check what share of crawls return non-200 responses — redirects and 404s consume budget and yield nothing.
- 4Look at server logs for the most-crawled URLs. If parameter or filter URLs dominate, that is the waste.
- 5Check whether new pages get indexed promptly. If they do, crawl budget is not constraining you regardless of the numbers.
What actually recovers crawl
Crawl-delay in robots.txt is not on this list because Google ignores it. Crawl rate is managed by Google based on your server's response, and the way to raise it is to respond faster.
| Action | Effect | Effort |
|---|---|---|
| Remove faceted URL space | Large | Medium |
| Fix redirect chains | Medium | Low |
| Speed up server response | Medium — raises the crawl rate | Varies |
| Remove or noindex thin pages | Medium | Medium |
| Accurate sitemap with real lastmod | Small but free | Low |
| Block internal search results | Large if they were crawlable | Low |
What does not help
Submitting sitemaps repeatedly, requesting indexing in bulk, and pinging services do not increase crawl budget. Neither does adding more pages, which is the opposite. The lever is reducing waste and responding faster — everything else is activity rather than progress. See index bloat and crawl discovery for removing the waste and faceted navigation for the biggest single source of it.
How DomainLens contributes
DomainLens reports response time, redirect chains and indexability per URL — the three things that determine whether a fetch was worth Googlebot's time. For the catalogue-specific version see crawl budget for e-commerce .
- Does my site have a crawl budget problem?
- Probably not. Below a few thousand URLs it is almost never the constraint. If new pages get indexed promptly, budget is not your issue.
- Does crawl-delay in robots.txt work?
- Not for Google, which ignores it. Google manages crawl rate itself based on how your server responds.
- Do 404s waste crawl budget?
- Slightly. At scale it is worth cleaning up; on a small site the effect is negligible compared to other work.
- My pages are crawled but not indexed. Is that crawl budget?
- No — the opposite. Google spent budget on them and chose not to index. That is a content and quality question.