[WIP] Python modules in 2025
TL;DR
This is a working document. It is intended as a discussion around best practices / my own biases on setting a Python module from scratch.
Python version - 3.13
The lowest supported version is Python 3.9, which is the version that we should minimally support if we intend to release a publicly-usable module. Since this is intended for internal use, we can use the latest stable version of 3.13. Python 3.13 is the first Python version with experimental support for disabling the global interpreter lock (GIL).
Author’s note: Recently moved back to 3.12 due to uv support of pytorch.
Package manager - TBD
pip is the most widely used package manager. A common complaint, however, is poor dependency resolution.
conda is a batteries-included scientific Python distribution with reliable CUDA integration. This makes it particularly helpful when working with GPU-accelerated libraries like PyTorch.
uv is a more recent package manager that has been getting traction, written in Rust by Astral. It’s fully compatible with the pip ecosystem, but gets a speed up from parallel downloads of packages and aggressive caching of wheels.
[TBD] Modern containerization and deployment platforms like Docker and Modal provide robust environment isolation, enabling teams to use simpler package managers like pip or uv while adding conda only when needed for specific use cases.
CI/CD - Github Actions
Since our repository will be hosted on GitHub, a convenient starting point is to use GitHub Actions. This requires a workflow config file in .github/workflows. Actions can be triggered by repository events like pull requests.
Tests - pytest
pytest is a very popular option over the built-in unittest, due to its simpler syntax and powerful fixture system.
The author is partial to the use of the standalone mock package (vs unittest.mock and contextlib.ExitStack for mocks, but this is a personal preference.
Types - mypy
Perhaps a question worth asking is whether we need static type checking at all. It is the author’s view that static types help as guardrails or an additional check for expected behavior. In other words, ‘types are tests that you get for free’ or even perhaps ‘making illegal states unrepresentable’. LLMs also lower the toil of adding types.
mypy is the Guido-blessed type checker, here we can additional run the checks on strict mode to ensure types are added.
pyright is a commonly-used alternative that is used by VSCode’s LSP. In particular, pyright is able to run static type checks of both implementation (.py) and type stubs (.pyi), and is faster due to its TypeScript implementation. I don’t expect type stubs to an issue (at least until we start using protobufs say) and anecdotally find mypy errors simpler to parse, so my leaning is towards using mypy.
pydantic does runtime type checking. This is a nice-to-have, and should be invoked automatically where there’s test coverage. It’s a popular package that’s made even more popular for AI use cases by an AI Engineer conference talk.
Linter - ruff
ruff is a linter and formatter, also written in Rust by Astral. It is known for being fast.
Other commonly used tools here is flake8 for linting and black for formatting.
Deploys
As a starting point, we expect the private module to be used in Jupyter Notebook. This can be imported by providing GitHub API token on the import, exported as an environment variable from an .env file.
modal has gotten a lot of traction by making it easier to do ‘infrastructure as code’, but with Python instead of Terraform. This allows for simpler deploys (since the same code is used locally and in production) and scaling up, and is popular for AI usage due to H100 access and more recently, ability to use AWS credits.
Miscellaneous
fastapi is a popular alternative to flask and django, mainly due to performance. At the time of writing, fastapi has close to 80k GitHub stars (agreed not the best metric, but hey).
logfire is a logging solution developed by the pydantic team post VC-funding. The author has not used it but hears good things.
dlt is like open source FiveTran. The author first came across it through a Modal blog post, and more recently has attracted interest by making ‘portable data lakes’ possible by building on top of Apache Iceberg.
mojo is the Chris Lattner-created superset of Python, perhaps a re-channeling of Swift for ML but in Python syntax. It’s more likely we’ll use libraries built on top of mojo rather than direct imports, but it’s an exciting addition to the Python ecosystem.