AdvancedLesson 26 of 306 min read

Project structure, packaging and logging

Organise a Python project properly: folders, packages, __main__, argparse, pyproject.toml, environment settings and logging.

In this lesson you will

  • Lay out a project with src and tests
  • Build command-line tools with argparse
  • Describe a project in pyproject.toml
  • Use environment variables and logging

About 22 minutes

From a pile of scripts to a real project

Your project started as one file. Now it has six, a data folder, tests and a few packages it depends on. Where does each thing go? How does a teammate install it with one command? How do you keep secrets out of the code? Professional Python projects follow a few agreed conventions, and learning them saves you from the mess that every growing project eventually becomes.

A tidy project layout

expense-tracker/
|-- pyproject.toml        project name, version, dependencies
|-- README.md             what it is and how to run it
|-- .gitignore            files Git must ignore (.venv, __pycache__, .env)
|-- requirements.txt      pinned packages (optional with pyproject)
|-- src/
|   `-- tracker/
|       |-- __init__.py
|       |-- models.py     data classes
|       |-- storage.py    load and save
|       |-- cli.py        command-line interface
|       `-- __main__.py   makes "python -m tracker" work
`-- tests/
    |-- test_models.py
    `-- test_storage.py
  • src/package: your code lives in one package folder, so imports are consistent.
  • tests/: kept apart from the code.
  • README.md: the first thing anyone reads. Say what the project does, how to install it and a short usage example.
  • .gitignore: never commit .venv, __pycache__ or secrets.

Building the package for real

Here is a tiny package. Notice __main__.py: it lets anyone run the project with python -m greeter.

"""A tiny greeting package."""
__version__ = "0.1.0"

from .core import greet
def greet(name: str) -> str:
    return f"Hello, {name}!"
import sys
from .core import greet

def main() -> None:
    name = sys.argv[1] if len(sys.argv) > 1 else "world"
    print(greet(name))

if __name__ == "__main__":
    main()
$ python -m greeter Sara
Hello, Sara!

Command-line options with argparse

import argparse

def build_parser():
    parser = argparse.ArgumentParser(description="Add up some numbers.")
    parser.add_argument("numbers", nargs="+", type=float, help="numbers to add")
    parser.add_argument("--round", type=int, default=2, help="decimal places")
    parser.add_argument("-v", "--verbose", action="store_true", help="explain the result")
    return parser

def main(argv=None):
    args = build_parser().parse_args(argv)
    total = round(sum(args.numbers), args.round)
    if args.verbose:
        print(f"Adding {len(args.numbers)} numbers gives {total}")
    else:
        print(total)

if __name__ == "__main__":
    main()
$ python cli.py 1.5 2.25 3 -v --round 1 && python cli.py --help
Adding 3 numbers gives 6.8
usage: cli.py [-h] [--round ROUND] [-v] numbers [numbers ...]

Add up some numbers.

positional arguments:
  numbers        numbers to add

options:
  -h, --help     show this help message and exit
  --round ROUND  decimal places
  -v, --verbose  explain the result

argparse builds the --help text, checks types and reports errors for you. Positional arguments are required; options start with dashes.

pyproject.toml: describing your project

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "expense-tracker"
version = "0.1.0"
description = "A command-line expense tracker"
readme = "README.md"
requires-python = ">=3.10"
dependencies = ["requests>=2.30"]

[project.optional-dependencies]
dev = ["pytest", "mypy"]

[project.scripts]
expenses = "tracker.cli:main"

[tool.setuptools.packages.find]
where = ["src"]

This one file replaces the older setup.py. It names the project, lists its dependencies, and even creates a command: after installing, typing expenses in any terminal runs tracker.cli.main.

Installing and sharing

# inside your virtual environment, from the project folder
pip install -e .             # editable install: code changes apply immediately
pip install -e ".[dev]"      # also install the dev tools

pip install build twine
python -m build              # creates the files in dist/
twine upload dist/*          # publish to PyPI (needs an account)
Good to know

The commands above are shown as a recipe: they work on your own computer, in your own project. The examples with output elsewhere in this lesson were run for real.

Configuration and secrets

import os

DEBUG = os.environ.get("APP_DEBUG", "0") == "1"
API_KEY = os.environ.get("API_KEY")
DATA_DIR = os.environ.get("DATA_DIR", "data")

def describe():
    return {"debug": DEBUG, "has_key": API_KEY is not None, "data_dir": DATA_DIR}
$ python -c "import config; print(config.describe())" && APP_DEBUG=1 API_KEY=abc python -c "import config; print(config.describe())"
{'debug': False, 'has_key': False, 'data_dir': 'data'}
{'debug': True, 'has_key': True, 'data_dir': 'data'}

Read secrets and settings from environment variables. For local development, put them in a file called .env (listed in .gitignore) and load it with the python-dotenv package. The code never contains the secret itself.

Logging instead of print

import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(name)s | %(message)s")
log = logging.getLogger("tracker")

log.debug("hidden at INFO level")
log.info("Loaded 12 expenses")
log.warning("Data file is old")
try:
    1 / 0
except ZeroDivisionError:
    log.exception("Calculation failed")
$ python app.py
INFO | tracker | Loaded 12 expenses
WARNING | tracker | Data file is old
ERROR | tracker | Calculation failed
Traceback (most recent call last):
  File "./app.py", line 10, in <module>
    1 / 0
    ~~^~~
ZeroDivisionError: division by zero

print is for talking to the user. Logging is for talking to developers: it has levels (DEBUG, INFO, WARNING, ERROR), timestamps if you want them, and can go to a file. log.exception inside an except block records the full traceback.

A checklist before sharing a project

  1. A README with install and usage instructions.
  2. A .gitignore that excludes the virtual environment and secrets.
  3. Dependencies listed, with versions.
  4. Tests that pass with one command.
  5. A licence file if others may use it (for example MIT).
  6. No passwords or keys in the code or Git history.
  7. A version number you update with each release.
Common mistake

Committing secrets or the virtual environment to Git. Once a key is in the history it is exposed, even if you delete the file later. Create the .gitignore first, before your first commit.

Practice

Write an argparse parser for a tool greet.py that takes a required name and an optional --shout flag, then parse the list ["Sara", "--shout"] without using the real command line.

Hint: parse_args accepts a list, which is handy for testing.

Show one possible answer
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("name")
parser.add_argument("--shout", action="store_true")

args = parser.parse_args(["Sara", "--shout"])
message = f"Hello, {args.name}"
print(message.upper() if args.shout else message)
HELLO, SARA
Practice

Set up logging so messages go to a file called app.log at INFO level, write two messages, then print the file.

Show one possible answer
import logging

logging.basicConfig(filename="app.log", level=logging.INFO, format="%(levelname)s: %(message)s")
logging.info("Started")
logging.warning("Careful")
logging.shutdown()
print(open("app.log").read())
INFO: Started
WARNING: Careful
Copy a good project

Look at the structure of a respected open-source project on GitHub in a field you like. Seeing how experienced developers lay out their folders teaches you more than any rulebook.

Check what you learned

Five quick questions. Pick an answer and you will see straight away whether it is right and why.

What does __main__.py in a package allow?

  1. Faster imports
  2. Type checking
  3. Testing
  4. Running it with python -m package

python -m name runs that file.

Which file describes a modern Python project?

  1. pyproject.toml
  2. main.txt
  3. project.exe
  4. index.py

It holds name, version and dependencies.

Where should API keys live?

  1. In the README
  2. In test files
  3. In environment variables, not in code
  4. In comments

Keep secrets out of version control.

Why use logging instead of print for diagnostics?

  1. It is shorter
  2. It is required
  3. It is faster only
  4. It has levels and can go to files

Levels and handlers give you control.

What does pip install -e . do?

  1. Deletes it
  2. Installs your project in editable mode
  3. Publishes it
  4. Tests it

Changes to the code apply immediately.

Summary

  • Use a clear layout: src, tests, README, .gitignore.
  • argparse builds command-line tools.
  • Read settings and secrets from the environment.
  • Log with levels instead of print.

Next lesson

Next: async programming, doing many things at once without waiting.

Questions

Do I need pyproject.toml for a small script?

No. It matters once the project has several files, dependencies, or other people who will install it.

What is the difference between a library and an application?

A library is code others import. An application is a program people run, though many projects are both.

What does editable install mean?

Python links to your source folder, so edits take effect without reinstalling.

Progress is saved in this browser.