Server Logs, Googlebot Verification and Reporting
Read an access log, separate real Googlebot from impostors, spot wasted crawling and turn what you learn into a monthly report people actually use.
In this lesson you will
- Read an access log
- Verify Googlebot honestly
- Spot wasted crawling
- Build a monthly report
About 20 minutes
What did Googlebot really do?
Sofia’s hosting company sends her a file every month called the access log. She has never opened it. ‘It is just numbers,’ she says. But the log holds something no SEO tool can show as honestly: a line for every request that reached her server, including every visit from search engine crawlers. It answers questions such as ‘Which pages does Googlebot actually fetch?’, ‘How much of that effort goes on pages I do not care about?’ and ‘Are the visitors calling themselves Googlebot genuine?’ This lesson shows how to read a log with a few lines of code, and how to turn the results into a monthly report.
What a log line contains
Most web servers write one line per request in a standard layout. It holds the visitor’s address, the time, the request, the result code, the size and the visitor’s self-description, called the user agent. A typical line is long, so we will read only the parts that matter.
192.0.2.50 - - [03/Nov/2026:10:05:00 +0000] "GET /shop/?sort=price HTTP/1.1" 200 512 "-" "Mozilla/5.0 (compatible; Googlebot/2.1)"
The user agent is only a claim. Anybody can send ‘Googlebot’ as their name, and scrapers do. So a log tells you what visitors said they were, and you must check who they actually are.
Checking that a bot is real
Google documents two ways to verify a visitor claiming to be Googlebot. One is a reverse DNS lookup on the visitor’s address: the name that comes back should end in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name should return the original address. The other is to match the address against the lists of address ranges that Google publishes in data files. Both need live internet lookups, which a classroom demo cannot do, so the program uses a stand-in. It treats one range reserved for documentation, 192.0.2.0 to 192.0.2.127, as ‘Google’s’ network, and everything else as an impostor. In real life, use Google’s published method and ranges.
Read the log with code
The program below creates a sample log of 300 requests. All of it is made up, with documentation addresses, so it can show common patterns safely. It writes the log to a file, reads it back with a pattern, and answers four questions. The output is real, and we checked the counts again with the standard command-line tools grep and awk.
import ipaddress, random, re, subprocess
from collections import Counter
random.seed(5)
GOOGLE_NETS = [ipaddress.ip_network("192.0.2.0/25")] # stands in for Google's published ranges
paths = ["/", "/guides/", "/guides/soil/", "/shop/", "/shop/?sort=price", "/shop/?sort=name",
"/shop/?colour=red&sort=price", "/old-page/", "/about/"]
agents = {"google": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
"bing": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)",
"person": "Mozilla/5.0 (Windows NT 10.0) Chrome/120 Safari/537.36"}
lines = []
for n in range(300):
kind = random.choices(["google", "fake", "bing", "person"], [40, 8, 12, 40])[0]
ip = {"google": "192.0.2.%d", "fake": "203.0.113.%d", "bing": "198.51.100.%d", "person": "198.51.100.%d"}[kind] % random.randint(1, 120)
path = random.choices(paths, [8, 10, 10, 10, 14, 14, 10, 8, 6])[0] if kind in ("google", "fake") else random.choice(paths[:4] + paths[-1:])
status = 404 if path == "/old-page/" else 200
ua = agents["google"] if kind in ("google", "fake") else agents["bing" if kind == "bing" else "person"]
lines.append(f'{ip} - - [03/Nov/2026:10:{n % 60:02d}:00 +0000] "GET {path} HTTP/1.1" {status} 512 "-" "{ua}"')
open("access.log", "w").write("\n".join(lines) + "\n")
pat = re.compile(r'^(\S+) .*?"GET (\S+) HTTP/1.1" (\d{3}) \d+ "-" "(.*)"$')
rows = [pat.match(l).groups() for l in open("access.log").read().splitlines()]
def genuine(ip): return any(ipaddress.ip_address(ip) in net for net in GOOGLE_NETS)
claims = [r for r in rows if "Googlebot" in r[3]]
real = [r for r in claims if genuine(r[0])]
print(f"Log lines: {len(rows)}")
print(f"Claim to be Googlebot: {len(claims)}")
print(f"Pass the IP check: {len(real)} Fakes: {len(claims) - len(real)}")
params = [r for r in real if "?" in r[1]]
print(f"Real Googlebot hits on parameter URLs: {len(params)} of {len(real)} ({100 * len(params) // len(real)}%)")
print("Real Googlebot status codes:", dict(Counter(r[2] for r in real)))
print("Most requested by Googlebot:")
for path, n in Counter(r[1] for r in real).most_common(3):
print(f" {path}: {n}")
check = subprocess.run("grep -c Googlebot access.log", shell=True, capture_output=True, text=True).stdout.strip()
Log lines: 300
Claim to be Googlebot: 141
Pass the IP check: 127 Fakes: 14
Real Googlebot hits on parameter URLs: 55 of 127 (43%)
Real Googlebot status codes: {'200': 122, '404': 5}
Most requested by Googlebot:
/shop/?sort=price: 27
/: 17
/shop/?sort=name: 17
Here is how to read it. Of 300 lines, 141 claimed to be Googlebot, but only 127 passed the address check. The other 14 are impostors, and if you counted them in, you would think Google was visiting more often than it does. Among the genuine visits, 55 of 127, 43%, hit parameter URLs, the sort and filter pages from lesson 24. That is crawl effort spent on near-duplicates. The status codes show 122 successful requests and 5 not-found errors. The busiest address is /shop/?sort=price with 27 requests, and the next is /.
For a site this size, none of this is a crisis. Real logs on a large site would be millions of lines, and the same pattern, many requests to filter pages and errors, is exactly what you would then act on: tidy the links, block or canonicalise filters (lesson 24), and fix the 404s that crawlers keep hitting.
Search Console shows some of this too
Search Console’s Crawl Stats report summarises how Google crawls your site: requests over time, response codes, file types and the purpose of the crawl. It only covers Google, and only for your verified site, but it needs no server access. Logs add other crawlers, your own server’s view, and detail down to each address. A good habit is to compare the two. If your logs show far more ‘Googlebot’ visits than Crawl Stats, impostors are the likely reason.
A monthly report that people read
- Headline numbers. Clicks, impressions, CTR and indexed pages, this month against last month and the same month last year (lesson 17).
- What we did. The changes you made and their dates from your change log.
- What changed. Pages and groups that moved up or down, with a likely cause, said honestly. ‘We do not know yet’ is a fine answer.
- Technical health. Crawl errors, redirects, indexing problems and speed, from your audits.
- Next month. Three actions with owners and dates.
Keep it to one or two pages. Use a spreadsheet or a free dashboard tool that reads from Search Console, and send the same layout each month so readers learn where to look.
Trusting the user agent. A log full of ‘Googlebot’ may be half impostors. Verify by address before drawing conclusions about Google’s behaviour, and do not block a range of addresses that you have not checked.
Access logs contain visitors’ addresses, which can count as personal data in some places. Store logs securely, keep them only as long as you need to, and follow your local rules and your host’s policies.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What is a user agent?
- A server password
- A kind of redirect
- A visitor’s self-description, which can be faked
- A sitemap entry
It is only a claim. You must verify who a visitor really is.
How does Google say to verify Googlebot?
- Reverse then forward DNS, or its published ranges
- Trust the name
- Count the requests
- Check the file size
A reverse lookup must end in a Google domain and a forward lookup must return the same address, or match published ranges.
In the demo, why did the claim count differ from the pass count?
- Googlebot visited twice
- The log was empty
- Some visitors faked the Googlebot name
- Some pages were deleted
Impostors copied the user agent but failed the address check.
What does a high share of Googlebot hits on parameter URLs suggest?
- Crawl effort spent on near-duplicates
- A faster server
- More backlinks
- Better titles
Filter and sort pages repeat content, so crawl effort on them is largely wasted.
What should a monthly report include besides numbers?
- A guarantee of rankings
- What you did and what comes next
- Only screenshots
- Competitor passwords
Readers need the actions taken, honest explanations and the next steps.
Summary
- A log line records each request, including who claims to have made it.
- Verify Googlebot by DNS lookups or Google’s published ranges, not by name.
- Logs reveal crawl effort spent on parameter pages and errors.
- A clear monthly report states numbers, actions and next steps.
Questions
Can I get logs on shared hosting?
Often yes, through your host's control panel, though retention differs. If you cannot, use Search Console's Crawl Stats report, which needs no server access.
Do I need logs for a small site?
Usually not. They matter most for large sites, migrations and crawl problems. Learning to read them now makes you far more useful later.
Is it safe to block fake Googlebots?
Be careful. Verify the addresses first and block with your host or security tool, not with robots.txt, which only polite crawlers obey.
