AdvancedLesson 21 of 308 min read

Technical Audits at Scale and Crawl Budget

Crawl a whole test site, find patterns such as duplicate titles, filter URLs, thin pages and broken links, and learn honestly who needs to care about crawl budget.

In this lesson you will

  • Crawl a whole site
  • Spot site-wide patterns
  • Understand crawl budget honestly
  • Turn a report into actions

About 20 minutes

Fifty pages is easy. What about five thousand?

Noor looks after the website of a garden shop. It started with ten pages, and now it has categories, filters, hundreds of products and a blog. She can no longer check it by clicking around. ‘Is Google wasting its time on junk pages?’ she asks. ‘And would I even notice?’ This lesson is about auditing a whole site with a crawler: a program that behaves like a search engine, visits every page it can find and reports what it sees. You already ran small checks in earlier lessons. Now you will do it at scale.

Why audit the whole site

Checking pages one by one finds individual mistakes. Crawling finds patterns: a template that gives every product the same title, a filter that creates thousands of near-copies, a section that nothing links to. Patterns matter because one cause can spoil hundreds of pages at once, and one fix repairs them all.

What is crawl budget, and do you need to worry?

You will hear the term crawl budget: roughly, how many pages a search engine is willing and able to fetch from your site in a period. It depends on how fast your server responds and how much the engine wants your content. Here is the honest answer on whether it matters to you. Google says its crawl budget guide is an advanced one for very large sites, meaning roughly a million or more unique pages that change weekly, or about ten thousand or more pages that change daily, or sites with a large share of pages stuck in the ‘Discovered, currently not indexed’ state in Search Console. Google adds that most sites do not need to worry. For a smaller site, keeping the sitemap up to date and watching the indexing report is usually enough. Check Google’s current guide for exact wording.

So why learn this? Because the habits are the same at every size. A site with a thousand pages that wastes a quarter of its URLs on duplicates has a quality problem long before it has a budget problem, and a crawl audit finds both.

A test site and a real crawl

The program below builds a small garden shop on your own computer: a home page, four categories, seven products in each, three filter variations per category, an about page, one redirect and one broken link. It then crawls it from the home page exactly the way a search engine would, fetching each address once and following the links it finds. Everything it prints is measured. We also counted the requests on the server side and crawled the same site a second way, and the numbers matched.

import http.client, re, threading
from collections import Counter
from http.server import BaseHTTPRequestHandler, HTTPServer

def page(title, text, links):
    anchors = "".join(f'<a href="{u}">{u}</a> ' for u in links)
    return f"<html><head><title>{title}</title></head><body><h1>{title}</h1><p>{text}</p>{anchors}</body></html>"

LONG = "Hardy and easy to grow, chosen by our team after a full season of trials. " * 3
site, cats = {}, ["pots", "tools", "seeds", "soil"]
site["/"] = (200, None, page("Cedar Garden Shop", LONG, [f"/{c}/" for c in cats] + ["/about/", "/shop/"]))
site["/about/"] = (200, None, page("About Cedar Garden", LONG, ["/", "/gone/"]))
site["/shop/"] = (301, "/pots/", "")
for c in cats:
    items = [f"/{c}/item-{i}/" for i in range(1, 8)]
    facets = [f"/{c}/?sort=price", f"/{c}/?sort=name", f"/{c}/?sort=price&colour=blue"]
    site[f"/{c}/"] = (200, None, page(c.title() + " for the Garden", LONG, items + facets))
    for f in facets:
        site[f] = (200, None, page(c.title() + " for the Garden", LONG, items))
    for n, u in enumerate(items, 1):
        title = "Seeds" if c == "seeds" else f"{c.title()} item {n}"
        text = "Short note." if c == "soil" else LONG
        site[u] = (200, None, page(title, text, [f"/{c}/", "/"]))
sitemap = ["/", "/about/", "/old-sale/"] + [u for u in site if u.count("/") == 3]

hits = []
class Shop(BaseHTTPRequestHandler):
    def do_GET(self):
        hits.append(self.path)
        code, to, body = site.get(self.path, (404, None, "Not found"))
        self.send_response(code)
        if to: self.send_header("Location", to)
        self.end_headers(); self.wfile.write(body.encode())
    def log_message(self, *a): pass

server = HTTPServer(("127.0.0.1", 0), Shop)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

def get(path):
    c = http.client.HTTPConnection("127.0.0.1", port); c.request("GET", path)
    r = c.getresponse(); out = (r.status, r.getheader("Location"), r.read().decode()); c.close(); return out

seen, queue, info, broken = {}, [("/", 0, None)], {}, []
while queue:
    path, depth, src = queue.pop(0)
    if path in seen: continue
    status, to, body = get(path)
    seen[path] = status
    t = re.search(r"<title>(.*?)</title>", body)
    info[path] = (depth, t.group(1) if t else "", len(re.sub(r"<[^>]+>", " ", body).split()))
    if status == 404: broken.append((path, src))
    if status == 301: queue.append((to, depth + 1, path))
    for link in re.findall(r'href="([^"]+)"', body):
        queue.append((link, depth + 1, path))
server.shutdown()

ok = [p for p, s in seen.items() if s == 200]
params = [p for p in ok if "?" in p]
titles = Counter(info[p][1] for p in ok)
dupes = sum(n for t, n in titles.items() if n > 1)
thin = [p for p in ok if info[p][2] < 40]
print(f"Requests: {len(seen)}  (200: {len(ok)}, 301: {list(seen.values()).count(301)}, 404: {len(broken)})")
print(f"Parameter URLs: {len(params)} of {len(ok)} ({100 * len(params) // len(ok)}%)")
print(f"Pages sharing a title: {dupes}")
print(f"Thin pages (under 40 words): {len(thin)}")
print("Deepest page:", max(i[0] for i in info.values()), "clicks")
print("In sitemap, never found:", [u for u in sitemap if u not in seen])
print("Broken link:", broken[0][0], "from", broken[0][1])
Requests: 48  (200: 46, 301: 1, 404: 1)
Parameter URLs: 12 of 46 (26%)
Pages sharing a title: 23
Thin pages (under 40 words): 7
Deepest page: 2 clicks
In sitemap, never found: ['/old-sale/']
Broken link: /gone/ from /about/

Read the report from the top. The crawler made 48 requests. 46 returned a normal page, one was a redirect and one was a not-found error. That fits what we built.

Now the interesting lines. 12 of the 46 pages, 26%, are parameter URLs, the filter and sort variations such as /pots/?sort=price. They show the same products as the main category page. On a small site that costs nothing, but imagine the same ratio on a site ten thousand times larger. The report also found 23 pages sharing a title: the category pages and their filter copies all have one title, and seven seed products share the single title ‘Seeds’. And 7 pages are thin, with under 40 words, the soil products.

Last, two connection problems. The page /old-sale/ is listed in the sitemap but the crawler never reached it, so no link points to it. And /gone/, linked from /about/, returns a not-found error. The deepest page was only 2 clicks from the home page, which is healthy.

From report to action

  • Duplicates from filters: decide which filter pages deserve to exist, and keep the rest from being crawled or point them at the main page (lesson 24).
  • Shared titles: fix the template so each product gets a title built from its own name.
  • Thin pages: improve, merge or remove them (lesson 26).
  • Sitemap but unlinked: link the page properly or take it out of the sitemap.
  • Broken link: fix the link or restore the page.

Real audits use a crawler tool, and several free and paid ones exist. The one in this lesson is a teaching model: it fetches pages one at a time, never runs JavaScript (lesson 22 covers that) and does not check images or speed. Use the output as a way to understand what a bigger tool does for you.

Crawling too hard

Crawling a live site too fast. A crawler that fires hundreds of requests a second can slow a small server down. Set a delay, start with a small number of pages, and crawl only sites you own or have permission to test.

An approximation

Search engines and your own crawler never see exactly the same site, because engines decide for themselves what to fetch. Treat your crawl as a close approximation and compare it with Search Console’s indexing and Crawl Stats reports.

Crawl your own site with a free crawler, or adapt this program. List the five patterns it finds, such as shared titles, parameter URLs, thin pages, unlinked pages and broken links. For each, write down the one template or setting that causes it.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does a crawl audit find that page-by-page checks often miss?

  1. Typing errors only
  2. Patterns across many pages
  3. Colour problems
  4. Server prices

One template fault can affect hundreds of pages, and a crawl shows that pattern at once.

According to Google, who needs to worry most about crawl budget?

  1. Every new blog
  2. Only sites with video
  3. Only shops with ads
  4. Very large or fast-changing sites

Google says most sites do not need to worry, and its guide is aimed at very large sites.

Why were the parameter URLs a concern in the demo?

  1. They repeat the same products
  2. They are always blocked
  3. They load faster
  4. They have no links

Filter and sort copies show the same content again, which multiplies URLs without adding value.

What does it mean that /old-sale/ was in the sitemap but never found?

  1. It is the fastest page
  2. It is indexed twice
  3. No page links to it
  4. It has a canonical tag

The crawler only follows links, so an unlinked page can sit in the sitemap and still be missed.

Why should you crawl gently?

  1. Search engines forbid it
  2. It deletes pages
  3. It changes titles
  4. A fast crawler can slow your server

Too many requests at once can overload a small server, so use delays and permission.

Summary

  • A crawl audit finds patterns that page-by-page checks miss.
  • Crawl budget matters mainly for very large or fast-changing sites.
  • Filter URLs, shared titles, thin pages and orphans are common findings.
  • Crawl gently and only sites you may test.

Next lesson

Lesson 22 looks at JavaScript and what search engines can actually see.

Questions

Do I need to buy a crawler tool?

No. Several free options exist for small sites, and you can write a simple one as in this lesson. Paid tools help with very large sites and reporting.

How often should I crawl my site?

After any big change, and on a regular schedule such as monthly. Compare each report with the last one to see what changed.

Is my small site affected by crawl budget?

Almost certainly not in a way you need to manage. Focus on quality, links and clean structure, and use Search Console to confirm your pages are indexed.

Progress is saved in this browser.