IntermediateLesson 12 of 206 min read

Testing Titles and Thumbnails: Clicks, Watch Time and Not Stopping Early

Compare two thumbnails by click rate and watch time share, run an exact test on the gap, and see why small tests mislead.

In this lesson you will

  • Explain how a title or thumbnail test works
  • Compare click rate with watch time share
  • Read an exact test for a click gap
  • Avoid stopping a test early

About 30 minutes

Which thumbnail is better?

Kofi makes two thumbnails for his basil video. Thumbnail A shows a wilted plant and big red words. Thumbnail B shows a calm hand pinching a leaf, with fewer words. Abena prefers A. A friend prefers B. Instead of arguing, they decide to test. A test shows each version to different viewers at the same time and compares what happens. This lesson explains how to read such a test, and how not to fool yourself.

What the site offers

At the time of writing, YouTube Studio has a feature, called Test and compare in some places and A/B test titles and thumbnails in others, that lets eligible creators compare up to three titles, thumbnails, or combinations on one video. The help pages explain that the site shows the options to different viewers and judges them by watch time share, which is the share of the watch time in the test that came from viewers who saw each option, and not by click rate alone. Results are reported with labels such as a winner, a preferred option, or no clear difference. Features, names and eligibility change, so read YouTube Help for what you can use.

Why watch time and not clicks? A picture that promises something the video does not give may get many clicks and then lose the viewer in seconds. YouTube’s own guidance notes that tools which only look at click rate may pick a different winner. The numbers below show how that can happen.

from math import comb
def fisher(a_clicks, a_n, b_clicks, b_n):
    clicks = a_clicks + b_clicks
    total = a_n + b_n
    denom = comb(total, a_n)
    def prob(k):
        return comb(clicks, k) * comb(total - clicks, a_n - k) / denom
    observed = prob(a_clicks)
    lo = max(0, a_n - (total - clicks))
    hi = min(a_n, clicks)
    return sum(prob(k) for k in range(lo, hi + 1) if prob(k) <= observed * (1 + 1e-9))
def wilson(k, n):
    z = 1.96
    p = k / n
    c = (p + z * z / (2 * n)) / (1 + z * z / n)
    h = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5) / (1 + z * z / n)
    return round((c - h) * 100, 1), round((c + h) * 100, 1)
a = {"impressions": 4000, "clicks": 248, "avg_minutes": 2.1}
b = {"impressions": 4000, "clicks": 200, "avg_minutes": 3.4}
for name, v in (("A", a), ("B", b)):
    print(name + ": CTR " + str(round(v["clicks"] / v["impressions"] * 100, 1)) + ", range " + str(wilson(v["clicks"], v["impressions"])))
p = fisher(a["clicks"], a["impressions"], b["clicks"], b["impressions"])
print("chance of this CTR gap by luck:", round(p, 3))
watch = {n: v["clicks"] * v["avg_minutes"] for n, v in (("A", a), ("B", b))}
whole = sum(watch.values())
for n in watch:
    print(n + ": watch minutes " + str(round(watch[n])) + ", share " + str(round(watch[n] / whole * 100, 1)) + " percent")
print("CTR winner:", "A" if a["clicks"] > b["clicks"] else "B")
print("watch time share winner:", max(watch, key=watch.get))
early = fisher(12, 200, 10, 200)
print("same rates after 200 impressions each, chance by luck:", round(early, 2))
A: CTR 6.2, range (5.5, 7.0)
B: CTR 5.0, range (4.4, 5.7)
chance of this CTR gap by luck: 0.022
A: watch minutes 521, share 43.4 percent
B: watch minutes 680, share 56.6 percent
CTR winner: A
watch time share winner: B
same rates after 200 impressions each, chance by luck: 0.83

Thumbnail A was shown 4,000 times and clicked 6.2 percent of the time. Thumbnail B was also shown 4,000 times and clicked 5.0 percent. The ranges, using the method from the data analysis course, are (5.5, 7.0) and (4.4, 5.7), which overlap only slightly. The program also runs an exact test for the gap in clicks. It counts every way the total clicks could have fallen between the two groups by chance, and finds that a gap this large would happen about 0.022 of the time, a little over two percent. So A really does earn more clicks.

Now watch time. The viewers of A watched an average of 2.1 minutes and the viewers of B watched 3.4. Multiplying clicks by average minutes gives the watch time each thumbnail brought: 521 minutes for A and 680 for B. As shares of the total, A gets 43.4 percent and B gets 56.6 percent. So the click winner is A, and the watch time share winner is B. The calm thumbnail attracted fewer clicks but the right viewers, the ones who stayed. A test that looked only at clicks would have chosen the wrong picture.

Do not stop early

The last line of the program shows the other trap. Suppose the same click rates, about 6 and 5 percent, had come from only 200 impressions each. The exact test says that a gap like that would appear by luck about 0.83 of the time, in most samples. With small numbers, almost any result is possible. This is why YouTube runs tests for days and why you should not end a test, or change a thumbnail by hand, the moment one side looks ahead.

Make a test worth running

  • Make the options truly different. Three versions of the same idea with a different shade of colour tell you nothing. Try different reasons to click.
  • Change one kind of thing at a time: test the thumbnail or the title, not both, if you want to learn which mattered.
  • Keep every option honest. A picture that misleads can win clicks and still lose the test, and it damages trust.
  • Write down what you learned. “Calm hands beat a wilted plant” is a lesson that carries to the next video.
  • Wait for the result. Judging early mixes luck with effect.
Common mistake: clicks alone, too soon

Picking the thumbnail with the best click rate after a day. A click rate says nothing about whether the viewer stayed. Look at watch time too, and wait for enough impressions.

Simplified and invented

The impressions, clicks and average minutes are made up. The watch-time calculation is a simplified version of what the site does. The exact test is a standard method, shown for teaching, and is not YouTube’s own formula.

Make two thumbnails for one video idea that give different reasons to click. Show them to five people and ask what they expect the video to be about.

Practice for this lesson

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does YouTube judge a title or thumbnail test by, according to its help pages?

  1. The number of likes
  2. Watch time share, not click rate alone
  3. The number of comments
  4. The colour of the picture

A click rate alone can pick a misleading picture.

Why can the click winner differ from the watch time winner?

  1. Clicks and watch time are the same thing
  2. Watch time ignores viewers
  3. A picture can attract clicks from viewers who then leave quickly
  4. The test is random

Thumbnail B had fewer clicks but longer views.

What does the exact test count?

  1. Every way the clicks could have fallen between the groups by chance
  2. The number of colours
  3. The length of the video
  4. The number of subscribers

It tells you how often a gap this big would happen by luck.

Why not stop a test after 200 impressions each?

  1. The site forbids it
  2. Tests need exactly 200
  3. With small numbers, almost any result can happen by luck
  4. Small tests are always right

The same rates gave a very high chance of luck.

What makes a test worth running?

  1. Options that are truly different and honest
  2. Three nearly identical pictures
  3. Misleading text
  4. Changing everything at once

Give viewers different reasons to click.

Summary

  • Tests show options to different viewers at the same time.
  • Watch time share can pick a different winner from clicks.
  • Small samples give luck, not evidence.
  • Test different ideas, keep them honest, and write down what you learn.

Next lesson

Lesson 13 plans series and playlists, and a calendar you can keep.

Questions

Do I need the built-in test feature?

It is convenient, if you are eligible, because it shows the options at the same time and uses watch time share. Without it, you can still learn by comparing similar videos carefully, but results are harder to trust.

How long should a test run?

Long enough for a good number of impressions. The help pages say it depends on the video, so follow the feature's own guidance and avoid changing things by hand in the middle.

Should I test every video?

No. Test when a video has enough impressions to give a clear answer and when you have two or three really different ideas. Keep the lessons in a notebook and apply them to new videos.

Progress is saved in this browser.