Best practices for structuring medium‑scale Python projects and testing

10 Eki 20:46

For a codebase that lives between a single script and a full‑blown service, I usually start with a **src‑layout** that mirrors the domain rather than the technical layers. Put everything that ships in a top‑level `src/` directory, then create one package per feature (or bounded context) inside it, e.g.: ``` src/ ├── myapp/ │ ├── __init__.py │ ├── auth/ │ │ ├── __init__.py │ │ ├── service.py │ │ └── models.py │ ├── billing/ │ │ ├── __init__.py │ │ ├── service.py │ │ └── utils.py │ └── common/ │ ├── __init__.py │ ├── logging.py │ └── validators.py tests/ ├── auth/ │ └── test_service.py ├── billing/ │ └── test_service.py └── conftest.py ``` Feature packages own their public API, while a small `common/` (or `utils/`) module houses truly cross‑cutting concerns—logging, small helpers, shared validators. Keep that common area minimal; if you find yourself pulling a function into `common` just because two features need it, consider extracting a dedicated library instead. This layout scales well for a 3‑5 person team because each developer can own a feature package without stepping on each other’s toes, yet the overall repo stays cohesive. On the testing side, I gravitate toward **pytest** with the built‑in `pytest-mock` fixture for simple mocking, and `responses` or `httpx-mock` when you need to stub HTTP calls. Keep tests next to the feature they exercise (as shown above) or under a parallel `tests/` tree; both work, but the parallel tree makes it easier to enforce a flat import path (`import myapp.auth.service as auth`). Aim for a **unit‑to‑integration ratio** of about 70 % unit, 30 % integration; the integration tests should spin up only the minimal external services (e.g., a real database via Docker Compose) and run against the same entry point you use in production. Type hints are now mature enough to be **opt‑in** without hurting CI speed. Run `mypy --strict` locally or as a pre‑commit hook; in CI you can lower the strictness (e.g., `--ignore-missing-imports`) to keep the pipeline fast while still catching obvious regressions. Pair that with **ruff** (or `flake8` + `black`) for linting and auto‑formatting; ruff is especially fast and can enforce both style and some static‑analysis rules in a single pass. A typical CI stage looks like: 1. `ruff check . && ruff format --check .` 2. `mypy src/` 3. `pytest -q --cov=src` If you need runtime validation (e.g., user‑supplied config), use **pydantic** or **attrs** with validators. They give you type‑safe models and raise clear errors at startup, so you don’t have to sprinkle manual checks throughout the code. For configuration and secrets, I separate **static config** (environment‑agnostic values) from **runtime secrets**. Store static config in a `config.yaml` or `.env` file that you load via `python‑dotenv` or `pydantic-settings`. Secrets (API keys, DB passwords) never live in the repo; instead, inject them through environment variables or a secret manager (e.g., HashiCorp Vault, AWS Secrets Manager) in the CI/CD pipeline. In development you can use a `.env.local` that is git‑ignored, and in CI you add the same variables via the pipeline’s secret store. This keeps the codebase clean and makes the same loading logic work across dev, test, and production environments.