BeginnerLesson 1 of 307 min read

What SEO Is and How Search Engines Find Your Pages

Understand crawling, indexing and ranking with a tiny search engine you can read, then set up the free tools every beginner needs.

In this lesson you will

  • Explain SEO in plain words
  • Describe crawl, index and rank
  • Run a toy search engine
  • Set up free tools

About 15 minutes

A bakery nobody can find

Maya bakes sourdough bread in Leeds. Last month she built a website with her recipes, photos and opening times. She is proud of it. Then a friend searched for ‘sourdough bread Leeds’, scrolled through the results, and could not find her anywhere. Maya’s first thought is, ‘Is Google ignoring me?’ The truth is more useful than that. Google probably does not know her pages exist yet, or does not yet understand what they are for. Fixing that is what this whole course is about.

What SEO means

SEO stands for search engine optimisation. It means making your website easy for search engines to find and understand, and easy for people to use once they arrive. Notice the two audiences. A search engine is a program, but the program exists to send people to pages they will be glad they visited. If you help the person, and also make the page clear to the program, you are doing SEO well.

SEO is different from paid advertising. Ads are results a business pays for, and they stop when the money stops. SEO results are called organic, because nobody pays for each click. Organic results are not free in the sense that they cost nothing to earn: they cost time, care and good content. They also take time to appear, and nobody can honestly promise you a ranking or a date. Treat any person who does as a warning sign.

The three jobs: crawl, index, rank

Every search engine does three jobs, in this order. Learn these three words now, because almost every SEO problem you meet will be a problem at one of these steps.

Job What happens What can go wrong
Crawl A program follows links from page to page and downloads what it finds. No link leads to the page, or you told the program to stay out.
Index The engine reads the page and stores what it is about in a giant database. The page is thin, a copy of another, or marked as not for the index.
Rank When someone searches, the engine picks pages from the index and puts them in order. Other pages answer the question better, or match what people want more closely.

Notice that ranking is the last step. A page that was never crawled cannot be indexed, and a page that was never indexed cannot rank. Beginners often worry about ranking when the real problem is that the page was never found.

See it happen: a tiny search engine

Real search engines are enormously more complicated, but the three jobs can be shown in a few lines of Python. Here is a toy site with five pages. Each page has some text and some links. The code crawls from the home page, builds an index of which words appear where, and then ranks pages for a search. You do not need to understand every line. Read the output, then read the explanation under it.

pages = {
 "/":        {"text": "maya's sourdough bakery fresh bread in leeds",
              "links": ["/bread", "/about"]},
 "/bread":   {"text": "sourdough bread recipe and how to bake sourdough bread at home",
              "links": ["/", "/starter"]},
 "/starter": {"text": "how to make a sourdough starter at home in seven days",
              "links": ["/bread"]},
 "/about":   {"text": "about maya who bakes bread every morning",
              "links": ["/"]},
 "/draft":   {"text": "old draft page about sourdough that nobody links to",
              "links": []},
}

# 1. CRAWL: start at the home page and follow links
seen, queue = [], ["/"]
while queue:
    url = queue.pop(0)
    if url in seen:
        continue
    seen.append(url)
    queue.extend(pages[url]["links"])
print("Crawled:", seen)
print("Never found:", [u for u in pages if u not in seen])

# 2. INDEX: remember which page contains which word
index = {}
for url in seen:
    for word in pages[url]["text"].split():
        index.setdefault(word, {})
        index[word][url] = index[word].get(url, 0) + 1

# 3. RANK: score pages for a search and sort them
def search(query):
    scores = {}
    for word in query.split():
        for url, count in index.get(word, {}).items():
            scores[url] = scores.get(url, 0) + count
    return sorted(scores.items(), key=lambda s: (-s[1], s[0]))

results = search("sourdough bread")
for url, score in results:
    print(url, "score", score)
Crawled: ['/', '/bread', '/about', '/starter']
Never found: ['/draft']
/bread score 4
/ score 2
/about score 1
/starter score 1

Look at the first two lines. The crawler started at the home page and followed links, so it found 4 pages. The page called /draft was never found, because no other page links to it. It does contain the word ‘sourdough’, but it never made it into the index, so a search cannot return it. That is exactly how a real page can be invisible: nothing points to it.

Now look at the ranking. The page /bread comes first with a score of 4, because it uses both searched words twice each. The toy ranks by counting words, and that is the whole point of calling it a toy. Real engines weigh hundreds of signals and are trained to prefer pages that genuinely help. If you simply repeated ‘sourdough bread’ fifty times, a modern engine would not reward you. We will see why in lesson 5.

A toy, not the real thing

This code is a teaching model, not a copy of how Google or Bing work. Their real methods are not public in full and change often. What the model shows correctly is the order of the jobs: find pages, store them, then order them.

What you can and cannot control

You cannot control a search engine. You cannot make it visit on a schedule, and you cannot force a position. What you can control is whether your pages can be reached through links, whether each page is clear about its topic, whether it loads quickly on a phone, and whether it truly helps the person who lands on it. The next nine lessons cover each of these in turn, and the beginner project at the end brings them together.

Free tools to set up today

You do not need paid software to start. These free tools are enough for a beginner. Menu names and screens change from time to time, so follow the on-screen instructions when you set them up.

  1. Google Search Console. A free service from Google. After you prove you own your site, it shows how your pages appear in Google search, which searches brought visitors, and which pages have problems.
  2. Bing Webmaster Tools. The same idea for Bing. Bing results also power some other search products, so it is worth the few minutes.
  3. Your browser’s view-source. Right-click a page and choose to view the page source. You will see the title, headings and links exactly as a crawler sees them.
  4. A search like site:yourdomain.com. Typing this into Google gives a rough picture of which of your pages it knows about. It is approximate, not a full report, so use it as a quick sniff test only.
  5. A plain notes file. Write down today’s date, what you changed and why. SEO results take a while, and your notes tell you which change helped.
The launch-day trap

If your site runs on WordPress, check that the setting that asks search engines not to index the site is turned off after launch. Many sites are built with it switched on and then forgotten, which tells every crawler to stay away. The setting is usually found under the reading options, and its label can differ between versions, so look for wording about search engine visibility.

Open your own site (or any small site you like) and write down the three things a stranger would search to find it. Then, using view-source, find the page title. Does it mention any of your three searches? Do not change anything yet. Just notice. Lesson 4 will show you how to fix it.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What is the correct order of a search engine’s three jobs?

  1. Rank, crawl, index
  2. Crawl, index, rank
  3. Index, rank, crawl
  4. Crawl, rank, index

A page must be found (crawled), stored (indexed) and only then ordered (ranked).

In the toy engine, why was the /draft page never found?

  1. It had too few words
  2. It was too old
  3. Its text was too short
  4. No page linked to it

The crawler only follows links. With no link pointing at it, the page stayed invisible.

What does organic mean in SEO?

  1. Results earned without paying per click
  2. Results you pay for per click
  3. Results that cost nothing to earn
  4. Results that never change

Organic results are not bought click by click, but they still cost time and good work.

Which tool shows how your pages appear in Google search?

  1. A spreadsheet
  2. Your phone camera
  3. Google Search Console
  4. A calculator

Search Console is free and reports searches, clicks and problems for your own site.

Can anyone honestly guarantee you a top ranking by a certain date?

  1. Yes, if they use tools
  2. Yes, if you pay monthly
  3. Yes, for new sites
  4. No, nobody can promise that

Search engines are not controlled by site owners or agencies, so a guaranteed ranking is a warning sign.

Summary

  • SEO makes your site easy for search engines and people to understand.
  • Search engines crawl, then index, then rank, in that order.
  • A page with no links pointing to it may never be found.
  • Nobody can honestly promise a ranking or a date.

Next lesson

Lesson 2 looks at why people search and how to match their intent.

Questions

Is SEO free?

Organic traffic is not paid per click, but SEO costs time and effort, and you may pay for hosting, tools or help. Search Console and Bing Webmaster Tools are free.

How long does SEO take to work?

It varies a lot by site, topic and competition. New pages can take weeks or months to be found and to settle in the results, so keep notes and be patient.

Do I need to know how to code to learn SEO?

No. You will see a little code in this course because it lets us measure real things, but you can follow every idea without writing any yourself.

Progress is saved in this browser.