E-Commerce SEO: Faceted Navigation, Pagination and Product Markup
Count how filters multiply URLs, write and test robots rules that tame them, handle pagination safely and check product markup, with real calculations.
In this lesson you will
- Count filter URL combinations
- Write and test robots rules
- Handle pagination safely
- Check product markup
About 20 minutes
Fifty thousand pages from twelve
Tariq runs an online shoe shop with twelve categories. His developer proudly added filters so shoppers can pick a colour, a size, a material, a price band and a sort order. Three months later Tariq’s search report shows tens of thousands of pages ‘discovered’ but not indexed, and his server is busy all day with search engine visits. The filters did not add pages that anyone searches for. They added a huge number of near-copies. This lesson explains how that happens and what to do about it, and then covers the other online-shop topics: pagination and product markup.
The maths of filter URLs
Each filter can be off or set to one of its options, and filters combine. So the number of possible pages is the product of (options + 1) across all filters. Here is the calculation for Tariq’s shop, done by the program below. The numbers are exact.
import math, re
facets = {"colour": 6, "size": 5, "material": 4, "price band": 4, "sort order": 3}
per_category = math.prod(n + 1 for n in facets.values()) # each facet: one choice or none
print(f"URLs per category: {per_category:,}")
print(f"Across 12 categories: {12 * per_category:,}")
print("If only colour is allowed:", facets["colour"] + 1, "URLs per category")
RULES = ["Disallow: /*?*sort=", "Disallow: /*?*price=", "Allow: /*?colour=*$"]
def to_regex(rule):
path = rule.split(":", 1)[1].strip()
end = path.endswith("$")
return re.compile("^" + re.escape(path.rstrip("$")).replace(r"\*", ".*") + ("$" if end else ""))
def allowed(url, rules):
hits = [(len(r.split(":", 1)[1].strip()), r.startswith("Allow"))
for r in rules if to_regex(r).match(url)]
return max(hits)[1] if hits else True # longest path wins; a tie goes to Allow
tests = ["/shoes/", "/shoes/?colour=red", "/shoes/?sort=price", "/shoes/?colour=red&sort=price", "/shoes/?price=10-20"]
for u in tests:
print(f"{u}: " + ("crawlable" if allowed(u, RULES) else "blocked"))
better = RULES + ["Disallow: /*?colour=*&sort="] # a longer, more specific rule
print("After adding one longer rule:")
for u in tests[1:4]:
print(f"{u}: " + ("crawlable" if allowed(u, better) else "blocked"))
product = {"@context": "https://schema.org", "@type": "Product", "name": "Clay pot, 20 cm",
"offers": {"@type": "Offer", "price": "12.50", "priceCurrency": "GBP",
"availability": "https://schema.org/InStock"}}
def check_product(p):
bad = [f"missing {f}" for f in ("name", "offers") if f not in p]
o = p.get("offers", {})
if not re.fullmatch(r"\d+(\.\d{1,2})?", str(o.get("price", ""))): bad.append("price not a plain number")
if len(o.get("priceCurrency", "")) != 3: bad.append("currency not a 3-letter code")
return bad
print("Product markup:", check_product(product) or "no problems")
print("With price '12,50 GBP':", check_product({**product, "offers": {**product["offers"], "price": "12,50 GBP"}}))
URLs per category: 4,200
Across 12 categories: 50,400
If only colour is allowed: 7 URLs per category
/shoes/: crawlable
/shoes/?colour=red: crawlable
/shoes/?sort=price: blocked
/shoes/?colour=red&sort=price: crawlable
/shoes/?price=10-20: blocked
After adding one longer rule:
/shoes/?colour=red: crawlable
/shoes/?sort=price: blocked
/shoes/?colour=red&sort=price: blocked
Product markup: no problems
With price '12,50 GBP': ['price not a plain number']
The first three lines tell the story. With five filters of 6, 5, 4, 4 and 3 options, each category can produce 4,200 different URLs. Across twelve categories that is 50,400. Most of them are pointless to searchers: nobody searches for ‘shoes, size 7, leather, 50 to 75, sorted by price’. If only the colour filter is allowed to be indexed, you are down to 7 URLs per category.
What Google recommends
Google’s guide to faceted navigation says to decide first whether you need those URLs in search at all. If you do not, prevent crawling of them, for example with robots.txt rules, and note that using the nofollow attribute on the links does not by itself stop crawling. A canonical tag pointing at the main version can reduce crawling of the copies over time, but it is a hint rather than a block. If you do want some filtered pages indexed, such as a page for red running shoes that people really search for, give them clean, stable addresses and make sure each is a real, useful page. Read Google’s current guide, since it is updated.
Writing robots rules for parameters
Search engines such as Google understand two wildcard symbols in robots.txt rules: * matches any characters and $ marks the end of an address. When several rules match a page, Google applies the most specific one, which means the longest path, and if two rules tie it applies the less restrictive Allow. Tools sometimes count wildcards differently, so always test your file in Search Console before you publish it.
In the program, three rules block the sort and price filters but allow the colour filter. The test addresses show it working: /shoes/?sort=price is blocked and /shoes/?colour=red is crawlable. But look at the fourth test. / is crawlable, even though we meant to block sorting. The reason is that our Allow rule is longer than the Disallow rule, so it wins. This is a typical robots.txt bug and it is easy to miss by eye. The fix, in the second block of output, is a longer and more specific Disallow rule, which then takes precedence. Notice how the sorted colour page is now blocked and the plain colour page is still crawlable.
Pagination
Large categories are split into numbered pages. Google no longer uses the old rel=prev and rel=next hints for this, announced in 2019, so you do not depend on them. What matters is that each page in the series is a normal page with real links to the next and previous ones, so crawlers can reach every product. Give each page its own address and a self-referencing canonical tag. A common mistake is pointing all numbered pages’ canonical tags at page one, which tells engines to ignore the products on later pages.
Product markup
Product pages can use structured data (lesson 15) for the product name, price, currency and availability. The program’s last two lines check a tiny Product record. The price must be a plain number such as 12.50, and the currency a three-letter code such as GBP. Writing ‘12,50 GBP’ in the price field, as in the last line, is flagged. Details of which product features appear in results change, so check Google’s current product documentation.
Blocking filter pages in robots.txt and expecting them to disappear from the index. A blocked address can still be known to the engine from links. Use robots.txt to save crawling, and use noindex or canonical tags to manage what is indexed, as in lesson 7.
The rule tester in the program follows Google’s documented longest-match behaviour for teaching. It is not Google’s own parser. Use Search Console’s robots.txt tools to test your real file.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
How is the number of filter URLs calculated?
- Add all the options
- Count the categories
- Multiply (options + 1) for each filter
- Use the page count
Each filter can be unset or take one option, and they combine, so the counts multiply.
Why does nofollow on filter links not solve over-crawling?
- It deletes the pages
- It blocks images
- It speeds up the server
- It does not stop crawling by itself
Google says nofollow alone does not prevent crawling of those addresses.
When two robots rules match, which wins?
- The first in the file
- The one with the longer path
- The last in the file
- The shorter one
Google applies the most specific rule, measured by path length, and ties go to Allow.
Why was colour plus sort still crawlable in the demo?
- Robots files cannot block sort
- The test was wrong
- The Allow rule was longer than the Disallow
- Colour has no rule
The longer Allow rule took precedence, so a longer Disallow was needed.
What is a safe canonical setup for numbered category pages?
- Each page canonical to itself
- All pages canonical to page one
- No links between pages
- Block pages two onward
Pointing every page at page one tells engines to ignore later products.
Summary
- Filters multiply URLs, so most combinations should not be crawled or indexed.
- The longest matching robots rule wins and ties go to Allow, so test combinations.
- Keep numbered pages crawlable with self-referencing canonicals.
- Product markup needs plain-number prices and currency codes.
Questions
Should I index filtered pages?
Only the ones people really search for and that make useful pages, such as a popular colour or type. Keep the rest out of crawling or point them at the main page.
Is a canonical tag enough to control filter pages?
It is a hint and can reduce crawling over time, but it does not block crawling. If crawling is a problem, combine it with sensible robots rules and clean internal links.
Do I need Product markup?
It is optional but helps search engines understand your products and may make pages eligible for richer results. It must match what shoppers see on the page.
