BeginnerLesson 7 of 306 min read

Crawling and Indexing: robots.txt, Sitemaps and noindex

Learn the difference between crawling and indexing, read and write robots.txt, build a sitemap, and avoid the launch-day trap that hides sites.

In this lesson you will

  • Tell crawling from indexing
  • Read a robots.txt file
  • Build and check a sitemap
  • Run a launch-day checklist

About 17 minutes

Sara’s shop is invisible

Sara opened an online shop selling hand-made candles. Her web designer built the site at a test address and, sensibly, told search engines to stay away while it was being built. On launch day the shop moved to the real address, but that instruction moved with it. Three weeks later Sara’s site still does not appear anywhere. Nothing is wrong with her candles or her writing. A single forgotten instruction is blocking everything. This lesson explains the instructions you can give crawlers, and how to check them.

Crawling is not indexing

Two ideas are easy to confuse. Crawling is a program visiting your page. Indexing is the engine storing the page so it can appear in results. You can control these separately, with different tools, and mixing them up causes a lot of trouble.

  • robots.txt controls whether crawlers may visit. It is one file at the top of your site.
  • noindex tag controls whether a page may be stored. It sits in each page’s head section.
  • Sitemap is a list of pages you want found, in a file such as sitemap.xml.
  • Canonical tag names the main version of a page. It also sits in the head section.

robots.txt: a polite note on the door

The robots.txt file is a plain text file at the top level of your site. It tells well-behaved crawlers which areas they should not visit, for example your admin area or shopping cart. Each rule names a user agent (the crawler) and a path. The star means all crawlers. The file is public, so never use it to hide anything secret, because anyone can read it. It is a request that honest crawlers follow, not a lock.

A trap worth remembering

If you block a page in robots.txt, crawlers cannot visit it, so they cannot see a noindex tag on it either. A blocked page can sometimes still appear in results as a bare link, because other sites point to it. If you want a page kept out of the index, let it be crawled and give it a noindex tag instead. If you want crawlers to skip a section for efficiency, such as a cart, use robots.txt. Choose by what you actually want.

Sitemaps

A sitemap is a file listing the pages you want search engines to find, usually with the date each was last changed. It is a helpful hint, especially for new sites or pages with few links. It is not a command, and listing a page does not guarantee indexing. Only list pages you really want found: pages that load normally and are not marked noindex. Keep the dates honest.

Try it: check rules and build a sitemap

The first half of this program reads a robots.txt and asks whether four paths may be visited. The second half builds a small sitemap file and reads it back as a crawler would, checking every address and date. The domain example.com is a reserved name that exists for examples. The output is real.

from urllib.robotparser import RobotFileParser

robots = """User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://example.com/sitemap.xml
"""
rp = RobotFileParser()
rp.parse(robots.splitlines())
for path in ["/", "/blog/sourdough-starter", "/admin/login", "/cart/"]:
    verdict = "allowed" if rp.can_fetch("*", "https://example.com" + path) else "blocked"
    print(f"{path:<26} {verdict}")
print("Sitemap listed at:", rp.site_maps()[0].replace("https://example.com", ""))

import re
import xml.etree.ElementTree as ET

NS = "http://www.sitemaps.org/schemas/sitemap/0.9"
pages = [("https://example.com/", "2026-10-01"),
         ("https://example.com/blog/sourdough-starter", "2026-10-12"),
         ("https://example.com/about", "2026-09-20")]
urlset = ET.Element("urlset", xmlns=NS)
for loc, last in pages:
    url = ET.SubElement(urlset, "url")
    ET.SubElement(url, "loc").text = loc
    ET.SubElement(url, "lastmod").text = last
ET.indent(urlset)
xml = '<?xml version="1.0" encoding="UTF-8"?>\n' + ET.tostring(urlset, encoding="unicode")
open("sitemap.xml", "w").write(xml)
print("Wrote sitemap.xml,", len(xml.splitlines()), "lines")

# read it back like a crawler would, and check it
tree = ET.fromstring(xml.split("?>", 1)[1])
locs = [u.find(f"{{{NS}}}loc").text for u in tree]
dates = [u.find(f"{{{NS}}}lastmod").text for u in tree]
ok = all(l.startswith("https://") for l in locs) and \
     all(re.fullmatch(r"\d{4}-\d{2}-\d{2}", d) for d in dates)
print("URLs in sitemap:", len(locs), "| all https and dated:", ok)
/                          allowed
/blog/sourdough-starter    allowed
/admin/login               blocked
/cart/                     blocked
Sitemap listed at: /sitemap.xml
Wrote sitemap.xml, 15 lines
URLs in sitemap: 3 | all https and dated: True
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/sourdough-starter</loc>
    <lastmod>2026-10-12</lastmod>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-09-20</lastmod>
  </url>
</urlset>

Read the verdicts. The home page and the blog post are allowed. /admin/login and /cart/ are blocked, exactly as the two Disallow rules say. The program also found the sitemap address listed in the file, which helps crawlers discover it. In the second half, the program wrote the sitemap file you can see below the output. It holds 3 addresses, and the check confirms every one starts with https and every date is in the year-month-day form. The sitemap standard expects that date format.

Different readers, different details

Python’s built-in robots reader checks rules in the order they appear, while Google gives priority to the most specific matching rule. The sample file was written so that both approaches give the same answer. For real sites, check your file with Search Console, because complicated files can be read differently by different tools.

Check your own site

  1. Type your address followed by /robots.txt into a browser. Look for a line that says Disallow with just a slash. That blocks the whole site.
  2. View the source of your home page and search for the word noindex. If you find it in a robots meta tag on a page you want found, remove it.
  3. If you use WordPress, check the reading option that asks engines not to index the site is switched off.
  4. Make sure a sitemap exists, and submit its address in Search Console. Many site builders and SEO plugins create one for you.
Forgotten switches

Leaving a staging copy open to crawlers, or launching with the block still on. Both are easy to avoid with a launch checklist that includes these four checks.

Open your robots.txt and sitemap in a browser. Write down what each blocks or lists. Is there any page you want found that is blocked, or any page you want hidden that is listed?

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does robots.txt mainly control?

  1. Page colours
  2. Page titles
  3. Whether crawlers may visit
  4. Image size

It asks well-behaved crawlers to stay out of listed areas. It is about crawling, not indexing.

How do you keep a page out of the index reliably?

  1. Allow crawling and add noindex
  2. Block it in robots.txt only
  3. Delete its title
  4. Shrink its images

Crawlers must be able to visit the page to read its noindex tag.

Is robots.txt a good place to hide secret pages?

  1. Yes, it is private
  2. Yes, it is encrypted
  3. No, it is public to anyone
  4. No, it is blocked

Anyone can read it, so listing a secret path there actually advertises it.

What is a sitemap?

  1. A list of pages you want found
  2. A picture of your site
  3. A password file
  4. A speed test

It is a hint to engines about pages you want crawled, not a guarantee of indexing.

In the demo, which paths were blocked?

  1. / and /blog
  2. /admin/login and /cart/
  3. Only /cart/
  4. None of them

The two Disallow lines covered /admin/ and /cart/, so those paths were blocked.

Summary

  • Crawling is visiting; indexing is storing. Different tools control each.
  • robots.txt is public and only a request to crawlers.
  • To keep a page out of the index, allow crawling and use noindex.
  • A sitemap is a helpful hint, never a guarantee.

Next lesson

Lesson 8 covers mobile friendliness, page speed and image size.

Questions

Do I need a robots.txt file at all?

No. If there is none, crawlers assume they may visit everything. You only need one to ask crawlers to skip certain areas.

Will a sitemap make Google index my page?

It helps Google discover the page, but indexing depends on quality and other factors. Pages that are thin, copied or blocked may still be left out.

What is a canonical tag?

It is a line in a page that names the main address for that content when similar copies exist. It helps avoid duplicate versions competing with each other.

Progress is saved in this browser.