IntermediateLesson 18 of 305 min read

Comprehensions, generators and iterators

Write compact list, dictionary and set comprehensions, understand iterators, and use generators and the collections and itertools modules.

In this lesson you will

  • Write list, dict and set comprehensions
  • Use enumerate, zip, any and all
  • Understand iterators and write generators
  • Know Counter, defaultdict and itertools

About 22 minutes

Shorter, clearer, faster loops

You already know how to build a list with a loop: make an empty list, loop, append. It works, but takes three or four lines for a simple idea. Python has a compact notation for it called a comprehension, and an even smarter tool called a generator that produces values one at a time without storing them all. Professional Python code uses these constantly, and after this lesson you will read and write them with ease.

List comprehensions

# the long way
squares = []
for n in range(1, 6):
    squares.append(n * n)
print(squares)

# the comprehension way: [expression for item in iterable]
squares = [n * n for n in range(1, 6)]
print(squares)

names = ["sara", "ali", "mina"]
print([name.title() for name in names])
print([len(name) for name in names])
[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
['Sara', 'Ali', 'Mina']
[4, 3, 4]

Read it aloud: “n times n, for each n in the range”. The expression comes first, the loop second, all inside square brackets.

Adding a condition

numbers = range(1, 11)
evens = [n for n in numbers if n % 2 == 0]
print(evens)

words = ["apple", "kiwi", "banana", "fig"]
long_words = [w.upper() for w in words if len(w) > 4]
print(long_words)

labels = ["even" if n % 2 == 0 else "odd" for n in range(1, 6)]
print(labels)
[2, 4, 6, 8, 10]
['APPLE', 'BANANA']
['odd', 'even', 'odd', 'even', 'odd']
  • if at the end filters: it keeps or drops items.
  • if ... else at the start chooses between two values for every item.

Dictionary and set comprehensions

names = ["Sara", "Ali", "Mina"]
lengths = {name: len(name) for name in names}
print(lengths)

prices = {"pen": 45, "book": 320, "bag": 1250}
discounted = {item: price * 0.9 for item, price in prices.items()}
print(discounted)

swapped = {v: k for k, v in {"a": 1, "b": 2}.items()}
print(swapped)

unique_lengths = {len(w) for w in ["apple", "kiwi", "pear", "fig"]}
print(sorted(unique_lengths))
{'Sara': 4, 'Ali': 3, 'Mina': 4}
{'pen': 40.5, 'book': 288.0, 'bag': 1125.0}
{1: 'a', 2: 'b'}
[3, 4, 5]

Nested loops in a comprehension

pairs = [(x, y) for x in range(1, 4) for y in range(1, 3)]
print(pairs)

matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [n for row in matrix for n in row]
print(flat)
transposed = [[row[i] for row in matrix] for i in range(3)]
print(transposed)
[(1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2)]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[[1, 4, 7], [2, 5, 8], [3, 6, 9]]
Common mistake

Writing a comprehension so long and clever that nobody can read it. If it needs more than one or two clauses, or does not fit comfortably on a line, use an ordinary loop. Clarity beats cleverness every time.

Useful built-ins: enumerate, zip, any, all

names = ["Sara", "Ali", "Mina"]
marks = [90, 72, 85]

for i, name in enumerate(names, start=1):
    print(i, name)

for name, mark in zip(names, marks):
    print(f"{name}: {mark}")

print(dict(zip(names, marks)))
print(any(m < 50 for m in marks), all(m > 70 for m in marks))
print(sorted(marks, reverse=True), max(marks), min(marks))
1 Sara
2 Ali
3 Mina
Sara: 90
Ali: 72
Mina: 85
{'Sara': 90, 'Ali': 72, 'Mina': 85}
False True
[90, 85, 72] 90 72

Iterators: what a for loop really does

letters = ["a", "b", "c"]
it = iter(letters)
print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("No more items")
a
b
c
No more items

A for loop calls iter() on your list, then calls next() again and again until StopIteration. Anything you can loop over (lists, strings, files, dictionaries) is an iterable.

Generators: values on demand

def countdown(n):
    while n > 0:
        yield n               # hand back a value, then pause here
        n -= 1

for x in countdown(3):
    print(x)

gen = countdown(2)
print(next(gen), next(gen))
3
2
1
2 1

A function with yield is a generator. It does not run all at once: each time you ask for a value, it runs until the next yield, hands back the value, and pauses with its variables intact. Nothing is stored in a big list.

import sys

big_list = [n * n for n in range(100_000)]
big_gen = (n * n for n in range(100_000))      # round brackets = generator expression

print(type(big_gen).__name__)
print(sys.getsizeof(big_list) > 100_000, sys.getsizeof(big_gen) < 500)
print(sum(big_gen))
generator
True True
333328333350000
def fibonacci():
    a, b = 0, 1
    while True:                # an infinite sequence is fine: values are made on demand
        yield a
        a, b = b, a + b

from itertools import islice
print(list(islice(fibonacci(), 10)))
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
Good to know

A generator can be used only once. After it is exhausted, you need to create it again.

The collections module: handy containers

from collections import Counter, defaultdict, deque

votes = ["red", "blue", "red", "green", "blue", "red"]
print(Counter(votes).most_common(2))

groups = defaultdict(list)
for name, city in [("Sara", "Lahore"), ("Ali", "Karachi"), ("Mina", "Lahore")]:
    groups[city].append(name)
print(dict(groups))

queue = deque([1, 2, 3])
queue.append(4)
print(queue.popleft(), list(queue))
[('red', 3), ('blue', 2)]
{'Lahore': ['Sara', 'Mina'], 'Karachi': ['Ali']}
1 [2, 3, 4]

itertools: loop tools

from itertools import combinations, permutations, product, chain, groupby

print(list(combinations("ABC", 2)))
print(len(list(permutations("ABC"))))
print(list(product([1, 2], "ab")))
print(list(chain([1, 2], [3, 4])))

data = sorted(["apple", "avocado", "banana", "blueberry", "cherry"])
for letter, items in groupby(data, key=lambda w: w[0]):
    print(letter, list(items))
[('A', 'B'), ('A', 'C'), ('B', 'C')]
6
[(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]
[1, 2, 3, 4]
a ['apple', 'avocado']
b ['banana', 'blueberry']
c ['cherry']
Practice

From the list [3, 8, 15, 22, 7, 30], build a new list containing the squares of only the numbers above 10.

Show one possible answer
nums = [3, 8, 15, 22, 7, 30]
print([n * n for n in nums if n > 10])
[225, 484, 900]
Practice

Use a dictionary comprehension to count the length of each word in a sentence, mapping each word to its length.

Show one possible answer
sentence = "learn python every single day"
print({w: len(w) for w in sentence.split()})
{'learn': 5, 'python': 6, 'every': 5, 'single': 6, 'day': 3}
Practice

Write a generator evens(limit) that yields the even numbers up to the limit, then print them as a list for limit 10.

Hint: yield inside a for loop.

Show one possible answer
def evens(limit):
    for n in range(2, limit + 1, 2):
        yield n

print(list(evens(10)))
[2, 4, 6, 8, 10]
When to use which

Use a list comprehension when you need the whole result. Use a generator when you only loop once or the data is huge. Use a plain loop when the logic is complicated.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does [n * 2 for n in range(3)] give?

  1. [2, 4, 6]
  2. [0, 1, 2]
  3. [1, 2, 3]
  4. [0, 2, 4]

n takes 0, 1, 2 and is doubled.

Where does a filtering if go in a comprehension?

  1. At the start
  2. At the end
  3. Inside brackets only
  4. Nowhere

[x for x in items if condition].

What keyword makes a function a generator?

  1. return
  2. pause
  3. gen
  4. yield

yield hands back one value at a time.

Why use a generator for huge data?

  1. It does not store every value in memory
  2. It is shorter
  3. It sorts data
  4. It deletes data

Values are produced on demand.

What does zip(names, marks) do?

  1. Compresses files
  2. Sorts both
  3. Pairs items from both lists
  4. Joins strings

It walks through them together.

Summary

  • [expression for item in iterable if condition].
  • Dict and set comprehensions use braces.
  • yield makes a generator that produces values on demand.
  • collections and itertools save you writing loops.

Next lesson

Next: web requests and APIs, getting live data from the internet.

Questions

Are comprehensions faster than loops?

Usually a little, but the main benefit is readability. Do not sacrifice clarity for speed.

Why does my generator print nothing the second time?

A generator is used up after one pass. Create it again.

What is the difference between [ ] and ( ) comprehensions?

Square brackets build a list now. Round brackets make a generator that produces values later.

Progress is saved in this browser.