Comprehensions, generators and iterators
Write compact list, dictionary and set comprehensions, understand iterators, and use generators and the collections and itertools modules.
In this lesson you will
- Write list, dict and set comprehensions
- Use enumerate, zip, any and all
- Understand iterators and write generators
- Know Counter, defaultdict and itertools
About 22 minutes
Shorter, clearer, faster loops
You already know how to build a list with a loop: make an empty list, loop, append. It works, but takes three or four lines for a simple idea. Python has a compact notation for it called a comprehension, and an even smarter tool called a generator that produces values one at a time without storing them all. Professional Python code uses these constantly, and after this lesson you will read and write them with ease.
List comprehensions
# the long way
squares = []
for n in range(1, 6):
squares.append(n * n)
print(squares)
# the comprehension way: [expression for item in iterable]
squares = [n * n for n in range(1, 6)]
print(squares)
names = ["sara", "ali", "mina"]
print([name.title() for name in names])
print([len(name) for name in names])
[1, 4, 9, 16, 25]
[1, 4, 9, 16, 25]
['Sara', 'Ali', 'Mina']
[4, 3, 4]
Read it aloud: “n times n, for each n in the range”. The expression comes first, the loop second, all inside square brackets.
Adding a condition
numbers = range(1, 11)
evens = [n for n in numbers if n % 2 == 0]
print(evens)
words = ["apple", "kiwi", "banana", "fig"]
long_words = [w.upper() for w in words if len(w) > 4]
print(long_words)
labels = ["even" if n % 2 == 0 else "odd" for n in range(1, 6)]
print(labels)
[2, 4, 6, 8, 10]
['APPLE', 'BANANA']
['odd', 'even', 'odd', 'even', 'odd']
ifat the end filters: it keeps or drops items.if ... elseat the start chooses between two values for every item.
Dictionary and set comprehensions
names = ["Sara", "Ali", "Mina"]
lengths = {name: len(name) for name in names}
print(lengths)
prices = {"pen": 45, "book": 320, "bag": 1250}
discounted = {item: price * 0.9 for item, price in prices.items()}
print(discounted)
swapped = {v: k for k, v in {"a": 1, "b": 2}.items()}
print(swapped)
unique_lengths = {len(w) for w in ["apple", "kiwi", "pear", "fig"]}
print(sorted(unique_lengths))
{'Sara': 4, 'Ali': 3, 'Mina': 4}
{'pen': 40.5, 'book': 288.0, 'bag': 1125.0}
{1: 'a', 2: 'b'}
[3, 4, 5]
Nested loops in a comprehension
pairs = [(x, y) for x in range(1, 4) for y in range(1, 3)]
print(pairs)
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = [n for row in matrix for n in row]
print(flat)
transposed = [[row[i] for row in matrix] for i in range(3)]
print(transposed)
[(1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2)]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
[[1, 4, 7], [2, 5, 8], [3, 6, 9]]
Writing a comprehension so long and clever that nobody can read it. If it needs more than one or two clauses, or does not fit comfortably on a line, use an ordinary loop. Clarity beats cleverness every time.
Useful built-ins: enumerate, zip, any, all
names = ["Sara", "Ali", "Mina"]
marks = [90, 72, 85]
for i, name in enumerate(names, start=1):
print(i, name)
for name, mark in zip(names, marks):
print(f"{name}: {mark}")
print(dict(zip(names, marks)))
print(any(m < 50 for m in marks), all(m > 70 for m in marks))
print(sorted(marks, reverse=True), max(marks), min(marks))
1 Sara
2 Ali
3 Mina
Sara: 90
Ali: 72
Mina: 85
{'Sara': 90, 'Ali': 72, 'Mina': 85}
False True
[90, 85, 72] 90 72
Iterators: what a for loop really does
letters = ["a", "b", "c"]
it = iter(letters)
print(next(it))
print(next(it))
print(next(it))
try:
next(it)
except StopIteration:
print("No more items")
a
b
c
No more items
A for loop calls iter() on your list, then calls next() again and again until StopIteration. Anything you can loop over (lists, strings, files, dictionaries) is an iterable.
Generators: values on demand
def countdown(n):
while n > 0:
yield n # hand back a value, then pause here
n -= 1
for x in countdown(3):
print(x)
gen = countdown(2)
print(next(gen), next(gen))
3
2
1
2 1
A function with yield is a generator. It does not run all at once: each time you ask for a value, it runs until the next yield, hands back the value, and pauses with its variables intact. Nothing is stored in a big list.
import sys
big_list = [n * n for n in range(100_000)]
big_gen = (n * n for n in range(100_000)) # round brackets = generator expression
print(type(big_gen).__name__)
print(sys.getsizeof(big_list) > 100_000, sys.getsizeof(big_gen) < 500)
print(sum(big_gen))
generator
True True
333328333350000
def fibonacci():
a, b = 0, 1
while True: # an infinite sequence is fine: values are made on demand
yield a
a, b = b, a + b
from itertools import islice
print(list(islice(fibonacci(), 10)))
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
A generator can be used only once. After it is exhausted, you need to create it again.
The collections module: handy containers
from collections import Counter, defaultdict, deque
votes = ["red", "blue", "red", "green", "blue", "red"]
print(Counter(votes).most_common(2))
groups = defaultdict(list)
for name, city in [("Sara", "Lahore"), ("Ali", "Karachi"), ("Mina", "Lahore")]:
groups[city].append(name)
print(dict(groups))
queue = deque([1, 2, 3])
queue.append(4)
print(queue.popleft(), list(queue))
[('red', 3), ('blue', 2)]
{'Lahore': ['Sara', 'Mina'], 'Karachi': ['Ali']}
1 [2, 3, 4]
itertools: loop tools
from itertools import combinations, permutations, product, chain, groupby
print(list(combinations("ABC", 2)))
print(len(list(permutations("ABC"))))
print(list(product([1, 2], "ab")))
print(list(chain([1, 2], [3, 4])))
data = sorted(["apple", "avocado", "banana", "blueberry", "cherry"])
for letter, items in groupby(data, key=lambda w: w[0]):
print(letter, list(items))
[('A', 'B'), ('A', 'C'), ('B', 'C')]
6
[(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]
[1, 2, 3, 4]
a ['apple', 'avocado']
b ['banana', 'blueberry']
c ['cherry']
From the list [3, 8, 15, 22, 7, 30], build a new list containing the squares of only the numbers above 10.
Show one possible answer
nums = [3, 8, 15, 22, 7, 30]
print([n * n for n in nums if n > 10])
[225, 484, 900]
Use a dictionary comprehension to count the length of each word in a sentence, mapping each word to its length.
Show one possible answer
sentence = "learn python every single day"
print({w: len(w) for w in sentence.split()})
{'learn': 5, 'python': 6, 'every': 5, 'single': 6, 'day': 3}
Write a generator evens(limit) that yields the even numbers up to the limit, then print them as a list for limit 10.
Hint: yield inside a for loop.
Show one possible answer
def evens(limit):
for n in range(2, limit + 1, 2):
yield n
print(list(evens(10)))
[2, 4, 6, 8, 10]
Use a list comprehension when you need the whole result. Use a generator when you only loop once or the data is huge. Use a plain loop when the logic is complicated.
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What does [n * 2 for n in range(3)] give?
- [2, 4, 6]
- [0, 1, 2]
- [1, 2, 3]
- [0, 2, 4]
n takes 0, 1, 2 and is doubled.
Where does a filtering if go in a comprehension?
- At the start
- At the end
- Inside brackets only
- Nowhere
[x for x in items if condition].
What keyword makes a function a generator?
returnpausegenyield
yield hands back one value at a time.
Why use a generator for huge data?
- It does not store every value in memory
- It is shorter
- It sorts data
- It deletes data
Values are produced on demand.
What does zip(names, marks) do?
- Compresses files
- Sorts both
- Pairs items from both lists
- Joins strings
It walks through them together.
Summary
- [expression for item in iterable if condition].
- Dict and set comprehensions use braces.
- yield makes a generator that produces values on demand.
- collections and itertools save you writing loops.
Questions
Are comprehensions faster than loops?
Usually a little, but the main benefit is readability. Do not sacrifice clarity for speed.
Why does my generator print nothing the second time?
A generator is used up after one pass. Create it again.
What is the difference between [ ] and ( ) comprehensions?
Square brackets build a list now. Round brackets make a generator that produces values later.
