Project structure, packaging and logging
Organise a Python project properly: folders, packages, __main__, argparse, pyproject.toml, environment settings and logging.
In this lesson you will
- Lay out a project with src and tests
- Build command-line tools with argparse
- Describe a project in pyproject.toml
- Use environment variables and logging
About 22 minutes
From a pile of scripts to a real project
Your project started as one file. Now it has six, a data folder, tests and a few packages it depends on. Where does each thing go? How does a teammate install it with one command? How do you keep secrets out of the code? Professional Python projects follow a few agreed conventions, and learning them saves you from the mess that every growing project eventually becomes.
A tidy project layout
expense-tracker/
|-- pyproject.toml project name, version, dependencies
|-- README.md what it is and how to run it
|-- .gitignore files Git must ignore (.venv, __pycache__, .env)
|-- requirements.txt pinned packages (optional with pyproject)
|-- src/
| `-- tracker/
| |-- __init__.py
| |-- models.py data classes
| |-- storage.py load and save
| |-- cli.py command-line interface
| `-- __main__.py makes "python -m tracker" work
`-- tests/
|-- test_models.py
`-- test_storage.py
- src/package: your code lives in one package folder, so imports are consistent.
- tests/: kept apart from the code.
- README.md: the first thing anyone reads. Say what the project does, how to install it and a short usage example.
- .gitignore: never commit
.venv,__pycache__or secrets.
Building the package for real
Here is a tiny package. Notice __main__.py: it lets anyone run the project with python -m greeter.
"""A tiny greeting package."""
__version__ = "0.1.0"
from .core import greet
def greet(name: str) -> str:
return f"Hello, {name}!"
import sys
from .core import greet
def main() -> None:
name = sys.argv[1] if len(sys.argv) > 1 else "world"
print(greet(name))
if __name__ == "__main__":
main()
$ python -m greeter Sara
Hello, Sara!
Command-line options with argparse
import argparse
def build_parser():
parser = argparse.ArgumentParser(description="Add up some numbers.")
parser.add_argument("numbers", nargs="+", type=float, help="numbers to add")
parser.add_argument("--round", type=int, default=2, help="decimal places")
parser.add_argument("-v", "--verbose", action="store_true", help="explain the result")
return parser
def main(argv=None):
args = build_parser().parse_args(argv)
total = round(sum(args.numbers), args.round)
if args.verbose:
print(f"Adding {len(args.numbers)} numbers gives {total}")
else:
print(total)
if __name__ == "__main__":
main()
$ python cli.py 1.5 2.25 3 -v --round 1 && python cli.py --help
Adding 3 numbers gives 6.8
usage: cli.py [-h] [--round ROUND] [-v] numbers [numbers ...]
Add up some numbers.
positional arguments:
numbers numbers to add
options:
-h, --help show this help message and exit
--round ROUND decimal places
-v, --verbose explain the result
argparse builds the --help text, checks types and reports errors for you. Positional arguments are required; options start with dashes.
pyproject.toml: describing your project
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[project]
name = "expense-tracker"
version = "0.1.0"
description = "A command-line expense tracker"
readme = "README.md"
requires-python = ">=3.10"
dependencies = ["requests>=2.30"]
[project.optional-dependencies]
dev = ["pytest", "mypy"]
[project.scripts]
expenses = "tracker.cli:main"
[tool.setuptools.packages.find]
where = ["src"]
This one file replaces the older setup.py. It names the project, lists its dependencies, and even creates a command: after installing, typing expenses in any terminal runs tracker.cli.main.
Installing and sharing
# inside your virtual environment, from the project folder
pip install -e . # editable install: code changes apply immediately
pip install -e ".[dev]" # also install the dev tools
pip install build twine
python -m build # creates the files in dist/
twine upload dist/* # publish to PyPI (needs an account)
The commands above are shown as a recipe: they work on your own computer, in your own project. The examples with output elsewhere in this lesson were run for real.
Configuration and secrets
import os
DEBUG = os.environ.get("APP_DEBUG", "0") == "1"
API_KEY = os.environ.get("API_KEY")
DATA_DIR = os.environ.get("DATA_DIR", "data")
def describe():
return {"debug": DEBUG, "has_key": API_KEY is not None, "data_dir": DATA_DIR}
$ python -c "import config; print(config.describe())" && APP_DEBUG=1 API_KEY=abc python -c "import config; print(config.describe())"
{'debug': False, 'has_key': False, 'data_dir': 'data'}
{'debug': True, 'has_key': True, 'data_dir': 'data'}
Read secrets and settings from environment variables. For local development, put them in a file called .env (listed in .gitignore) and load it with the python-dotenv package. The code never contains the secret itself.
Logging instead of print
import logging
logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(name)s | %(message)s")
log = logging.getLogger("tracker")
log.debug("hidden at INFO level")
log.info("Loaded 12 expenses")
log.warning("Data file is old")
try:
1 / 0
except ZeroDivisionError:
log.exception("Calculation failed")
$ python app.py
INFO | tracker | Loaded 12 expenses
WARNING | tracker | Data file is old
ERROR | tracker | Calculation failed
Traceback (most recent call last):
File "./app.py", line 10, in <module>
1 / 0
~~^~~
ZeroDivisionError: division by zero
print is for talking to the user. Logging is for talking to developers: it has levels (DEBUG, INFO, WARNING, ERROR), timestamps if you want them, and can go to a file. log.exception inside an except block records the full traceback.
A checklist before sharing a project
- A README with install and usage instructions.
- A .gitignore that excludes the virtual environment and secrets.
- Dependencies listed, with versions.
- Tests that pass with one command.
- A licence file if others may use it (for example MIT).
- No passwords or keys in the code or Git history.
- A version number you update with each release.
Committing secrets or the virtual environment to Git. Once a key is in the history it is exposed, even if you delete the file later. Create the .gitignore first, before your first commit.
Write an argparse parser for a tool greet.py that takes a required name and an optional --shout flag, then parse the list ["Sara", "--shout"] without using the real command line.
Hint: parse_args accepts a list, which is handy for testing.
Show one possible answer
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("name")
parser.add_argument("--shout", action="store_true")
args = parser.parse_args(["Sara", "--shout"])
message = f"Hello, {args.name}"
print(message.upper() if args.shout else message)
HELLO, SARA
Set up logging so messages go to a file called app.log at INFO level, write two messages, then print the file.
Show one possible answer
import logging
logging.basicConfig(filename="app.log", level=logging.INFO, format="%(levelname)s: %(message)s")
logging.info("Started")
logging.warning("Careful")
logging.shutdown()
print(open("app.log").read())
INFO: Started
WARNING: Careful
Look at the structure of a respected open-source project on GitHub in a field you like. Seeing how experienced developers lay out their folders teaches you more than any rulebook.
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What does __main__.py in a package allow?
- Faster imports
- Type checking
- Testing
- Running it with python -m package
python -m name runs that file.
Which file describes a modern Python project?
- pyproject.toml
- main.txt
- project.exe
- index.py
It holds name, version and dependencies.
Where should API keys live?
- In the README
- In test files
- In environment variables, not in code
- In comments
Keep secrets out of version control.
Why use logging instead of print for diagnostics?
- It is shorter
- It is required
- It is faster only
- It has levels and can go to files
Levels and handlers give you control.
What does pip install -e . do?
- Deletes it
- Installs your project in editable mode
- Publishes it
- Tests it
Changes to the code apply immediately.
Summary
- Use a clear layout: src, tests, README, .gitignore.
- argparse builds command-line tools.
- Read settings and secrets from the environment.
- Log with levels instead of print.
Questions
Do I need pyproject.toml for a small script?
No. It matters once the project has several files, dependencies, or other people who will install it.
What is the difference between a library and an application?
A library is code others import. An application is a program people run, though many projects are both.
What does editable install mean?
Python links to your source folder, so edits take effect without reinstalling.
