AdvancedLesson 22 of 305 min read

Context managers and the with statement

Understand how with works and write your own context managers with __enter__/__exit__ and contextlib to guarantee clean-up.

In this lesson you will

  • Explain why with is safer than manual closing
  • Write a class-based context manager
  • Use @contextmanager for short managers
  • Use suppress and TemporaryDirectory

About 21 minutes

Always cleaning up after yourself

A good chef cleans the station when finished, even if a dish went wrong. Programs have the same duty: close the file, release the lock, disconnect from the database, restore the setting. Forgetting causes leaks, locked files and mysterious bugs. You already use the solution every time you write with open(...). This lesson explains how with works and shows you how to build your own.

Why try/finally is not enough

f = open("data.txt", "w")
try:
    f.write("hello")
finally:
    f.close()                  # runs even if write() fails

print(f.closed)

with open("data.txt", "w") as f2:   # the same safety in two lines
    f2.write("hello")
print(f2.closed)
True
True

What with really does

A context manager is any object with two special methods: __enter__, run when the block starts, and __exit__, run when the block ends, whether normally or because of an error.

class Announce:
    def __enter__(self):
        print("Setting up")
        return "the resource"

    def __exit__(self, exc_type, exc_value, traceback):
        print("Cleaning up")
        return False            # False: do not hide any error

with Announce() as thing:
    print("Working with", thing)

print("--- now with an error ---")
try:
    with Announce():
        print("About to fail")
        1 / 0
except ZeroDivisionError:
    print("The error still reached us")
Setting up
Working with the resource
Cleaning up
--- now with an error ---
Setting up
About to fail
Cleaning up
The error still reached us

Notice that “Cleaning up” printed even though the code inside crashed. That guarantee is the whole point. If __exit__ returns True, the error is swallowed; return False (or nothing) to let it continue.

A practical one: a timer

import time

class Timer:
    def __enter__(self):
        self.start = time.perf_counter()
        return self

    def __exit__(self, *exc):
        self.elapsed = time.perf_counter() - self.start
        return False

with Timer() as t:
    total = sum(range(500_000))

print(total)
print("Took less than 5 seconds:", t.elapsed < 5)
124999750000
Took less than 5 seconds: True

The easy way: contextlib.contextmanager

from contextlib import contextmanager

@contextmanager
def announce(name):
    print(f"Start {name}")
    try:
        yield name.upper()          # the with block runs here
    finally:
        print(f"Finish {name}")

with announce("backup") as label:
    print("Doing", label)
Start backup
Doing BACKUP
Finish backup

Code before yield is the setup; code after is the cleanup. Put the yield inside try/finally so cleanup runs even after an error. This style is far shorter than writing a class, so use it for most of your own managers.

Useful managers you can write in minutes

import os
from contextlib import contextmanager
from pathlib import Path

@contextmanager
def working_directory(path):
    old = os.getcwd()
    os.chdir(path)
    try:
        yield
    finally:
        os.chdir(old)            # always go back

Path("project").mkdir(exist_ok=True)
before = os.path.basename(os.getcwd()) != "project"
with working_directory("project"):
    inside = os.path.basename(os.getcwd())
after = os.path.basename(os.getcwd()) != "project"
print(before, inside, after)
True project True
from contextlib import contextmanager

@contextmanager
def temporary_setting(config, key, value):
    original = config[key]
    config[key] = value
    try:
        yield config
    finally:
        config[key] = original

settings = {"debug": False}
with temporary_setting(settings, "debug", True):
    print("Inside:", settings)
print("After:", settings)
Inside: {'debug': True}
After: {'debug': False}

Ready-made managers in the standard library

from contextlib import suppress, redirect_stdout
from pathlib import Path
import io, tempfile

# ignore one specific error on purpose
with suppress(FileNotFoundError):
    Path("does-not-exist.txt").unlink()
print("No crash")

# capture what a block prints
buffer = io.StringIO()
with redirect_stdout(buffer):
    print("this is captured")
print("Captured:", buffer.getvalue().strip())

# a folder that deletes itself
with tempfile.TemporaryDirectory() as folder:
    Path(folder, "scratch.txt").write_text("temp")
    print("Exists inside:", Path(folder).exists())
print("Exists after:", Path(folder).exists())
No crash
Captured: this is captured
Exists inside: True
Exists after: False

Several managers at once

with open("in.txt", "w") as f:
    f.write("alpha\nbeta\n")

with open("in.txt") as src, open("out.txt", "w") as dst:
    for line in src:
        dst.write(line.upper())

print(open("out.txt").read())
ALPHA
BETA
Common mistake

Using the resource after the with block has ended. The file is closed, the connection is gone. Do all the work inside the indented block, and keep only the results you need outside.

Practice

Write a context manager (with @contextmanager) called indent that prints “– start –” on entry and “– end –” on exit, even when the block raises an error. Test it with an error inside.

Show one possible answer
from contextlib import contextmanager

@contextmanager
def indent():
    print("-- start --")
    try:
        yield
    finally:
        print("-- end --")

try:
    with indent():
        print("working")
        raise RuntimeError("oops")
except RuntimeError:
    print("handled")
-- start --
working
-- end --
handled
Practice

Write a class-based context manager Counter whose __enter__ returns itself and whose __exit__ prints how many times add() was called inside the block.

Hint: *exc collects the three exception arguments.

Show one possible answer
class Counter:
    def __init__(self):
        self.n = 0
    def add(self):
        self.n += 1
    def __enter__(self):
        return self
    def __exit__(self, *exc):
        print("add() was called", self.n, "times")
        return False

with Counter() as c:
    c.add(); c.add(); c.add()
add() was called 3 times
Spot the pattern

Whenever you see a pair of actions that must always go together (open and close, lock and unlock, connect and disconnect, start and stop), that is a context manager waiting to be written.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

Which two methods make an object a context manager?

  1. __start__ and __stop__
  2. __enter__ and __exit__
  3. __init__ and __del__
  4. __open__ and __close__

with calls __enter__ then __exit__.

When does __exit__ run?

  1. Only on success
  2. Only on error
  3. Always, even after an error
  4. Never automatically

That guarantee makes cleanup reliable.

What does the code before yield in a @contextmanager function do?

  1. It is the setup
  2. It is the cleanup
  3. It is ignored
  4. It is an error

Setup before yield, cleanup after.

What does suppress(FileNotFoundError) do?

  1. Hides all errors
  2. Creates the file
  3. Ignores that one error inside the block
  4. Logs the error

Only the named error is ignored.

What happens if __exit__ returns True?

  1. The error is swallowed
  2. The program stops
  3. The block repeats
  4. Nothing

Returning True suppresses the exception.

Summary

  • with guarantees clean-up, even after errors.
  • __enter__ and __exit__ define a manager.
  • @contextmanager turns a generator into one.
  • Use managers for paired actions.

Next lesson

Next: type hints, making your code clearer and catching mistakes early.

Questions

Is with only for files?

No. It works for locks, database connections, temporary settings, timers and more.

Can __exit__ see the error?

Yes. It receives the exception type, value and traceback as arguments.

class or @contextmanager?

Use @contextmanager for short ones. Use a class when the manager needs lots of state or methods.

Progress is saved in this browser.