Guides
Log File Analysis for SEO: How Googlebot Actually Crawls Your Site
Search Console tells you what Google concluded. Server logs tell you what Google actually did — every request, with a timestamp and a status code.
Starte ein frisches DomainLens-Audit und nutze den Report als Prioritätenliste.
What server logs prove that no other tool can
A log line is a fact: this client requested this URL at this time and the server answered with this status. Nothing infers it, samples it, or summarises it. That makes logs the only place you can answer questions about crawl behaviour directly rather than by deduction.
Search Console reports Google's conclusions, aggregated and delayed. A crawler you run reports what a crawler would find. Neither tells you whether Googlebot fetched a specific URL last Tuesday, how often it returns, or how much of your crawl is being spent on URLs you never wanted indexed.
| Question | Logs | Search Console |
|---|---|---|
| Was this URL fetched, and when? | Yes, exactly | Last crawl date only, per URL |
| How often does Googlebot return? | Yes | No |
| Which URLs consume the crawl? | Yes | Aggregate stats only |
| What status did the bot receive? | Yes, per request | Sampled and delayed |
| Why is a page not indexed? | No | Yes — the coverage state |
Verify Googlebot before you analyse anything
The user agent string is trivially forged, and a meaningful share of traffic claiming to be Googlebot is not. Analysing unverified hits produces confident conclusions about a crawler that was never there.
- Accept only hostnames ending in googlebot.com, google.com, or googleusercontent.com.
- The forward lookup must return the original IP — a matching hostname alone is not enough.
- Google also publishes IP ranges as JSON, which is faster for bulk verification than per-line DNS.
- Cache verification results by IP; the same addresses recur constantly and DNS lookups dominate the runtime otherwise.
# 1. Reverse lookup the IP from the log line
host 66.249.66.1
# -> 1.66.249.66.in-addr.arpa domain name pointer
# crawl-66-249-66-1.googlebot.com.
# 2. Forward lookup that hostname
host crawl-66-249-66-1.googlebot.com
# -> crawl-66-249-66-1.googlebot.com has address 66.249.66.1
# It is genuine only if the hostname ends in googlebot.com
# or google.com AND resolves back to the original IP.The fields that carry the signal
A combined-format log line has more than you need. Five fields answer nearly every crawl question, and the rest is noise for this purpose.
- Split smartphone from desktop Googlebot — mobile-first indexing means the smartphone agent is the one that matters.
- Separate Googlebot from Google-InspectionTool and AdsBot; those are triggered by you or by ads, not by organic crawling.
- Keep the status code per request rather than per URL. The same URL returning 200 then 500 is the finding.
66.249.66.1 - - [14/Aug/2026:09:12:44 +0000] \
"GET /guides/technical-seo HTTP/1.1" 200 18422 \
"-" "Mozilla/5.0 (compatible; Googlebot/2.1; \
+http://www.google.com/bot.html)"
# ^IP ^timestamp ^method+URL ^status ^bytes
#
# IP -> verify it is really Google
# timestamp -> crawl frequency and recency
# URL -> what is consuming the crawl
# status -> what the bot actually received
# UA -> smartphone vs desktop GooglebotFour crawl patterns worth finding
Wasted crawl is the most common and the most fixable. If filtered and parameterised URLs dominate your Googlebot hits, the fix is upstream in faceted navigation , not in the logs. Sections that are never fetched usually point at orphan pages or links a crawler cannot follow.
| Pattern | What the logs show | What it means |
|---|---|---|
| Wasted crawl | Heavy hits on parameter or filter URLs | Crawl spent on pages you do not want indexed |
| Orphan crawling | Bot fetches URLs absent from your sitemap | Old URLs still linked or remembered |
| Ignored sections | Whole directories never fetched | Discovery or internal-linking failure |
| Error clusters | Repeated 5xx or 404 to the bot | Trust and crawl rate erode quietly |
A first pass you can run today
- 1Export at least thirty days of access logs — a week is too short to show frequency patterns.
- 2Filter to verified Googlebot hits only, then split smartphone from desktop.
- 3Rank URLs by hit count and read the top hundred. Wasted crawl is usually obvious immediately.
- 4Group by status code and investigate every 5xx and every 404 the bot received.
- 5Diff crawled URLs against your sitemap in both directions: crawled-but-not-listed, and listed-but-never-crawled.
- 6Re-run monthly and compare. The trend matters more than any single snapshot.
# Most-crawled URLs (verified-Googlebot lines only)
awk '{print $7}' googlebot.log | sort | uniq -c | sort -rn | head -50
# Status code distribution
awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn
# Crawl volume per day
awk -F'[][]' '{split($2,d,":"); print d[1]}' googlebot.log \
| sort | uniq -c
# Every URL that returned 5xx to the bot
awk '$9 ~ /^5/ {print $7}' googlebot.log | sort -uHow DomainLens contributes
Logs tell you which URLs Googlebot is spending its budget on; they cannot tell you whether those URLs deserve it. DomainLens answers the second half — status codes, canonical conflicts, robots directives, and indexability for the URLs your log analysis surfaces. Pair it with crawl budget for the strategy and crawl error reports for what Google reports back.
- How much log data do I need?
- Thirty days minimum. Shorter windows cannot distinguish a URL crawled rarely from one crawled once by chance, which is the distinction most conclusions depend on.
- Do small sites need log analysis?
- Rarely. Below a few thousand URLs, crawl budget is almost never the constraint. Logs become valuable when the site is large, heavily parameterised, or when pages are discovered but never indexed.
- Can I get logs from a CDN?
- Usually yes, and you often must — if a CDN serves cached responses, your origin logs never see those requests and will understate crawling significantly.
- Why does Googlebot crawl the same page repeatedly?
- Frequently changing pages, or pages with many internal links, get re-crawled more often. It becomes a problem only when that frequency crowds out URLs that are never fetched at all.