How To Make Python Dependencies Manageable
Cross-package dependencies are a nightmare. Every Readme says “install packages X, Y and Z” but they rarely mention the versions of those packages, let alone versions for their dependencies. Of course, the Readme was written three months ago, and the new version of package X relies on a dependency that’s also required by package Y. Unfortunately, they require different versions of that dependency. And then there’s your OS: it’s updated itself such that the old version of package X is incompatible, so now you have to use the new version of X. How are you supposed to get all this to work without a time machine?
If you work with Python, you’ll eventually run into projects with complex dependency trees like the above example. Since Python mostly hides these dependencies from you and automatically pulls them in as needed, it can be surprising to find out just how deep your dependencies go. As a result, you’re unlikely to have much insight into the the compatibility and/or conflicts amongst dependencies. Until it’s too late…
Approaches To Managing Python Dependencies
Luckily, you have a number of options you can explore to help you address these shortcomings, including:
Vendored dependencies — i.e. self-managed dependencies
- Pros: Guaranteed compatibility, availability and security; the best option if you’ve created custom patches for specific packages.
- Cons: Results in a larger repo since you have to host all the dependencies yourself, which also means you may have to create your own wheels. Also requires lots of extra work to perform updates, check for vulnerabilities, manage conflicts with system installs, etc.
- Verdict: This kind of approach is very labor intensive, and can be a distraction from just coding your app, slowing time to market. Avoid it if you can.
Requirements.txt/ Pipfile — the standard way to manage dependencies in Python
- Pros: shared source (PyPI) eliminates the need to manage each dependency yourself.
- Cons: Dependencies are constantly being updated, so discipline is required to make sure you’re pinning your package versions and creating a lockfile to prevent newer versions breaking your app.
- Verdict: A LOT less painful than vendored dependencies, but it still requires discipline.
Pre-built distributions — eg., Anaconda, or ActiveState’s Python distribution, ActivePython
- Pros: Requires no discipline since the vendor has already done the work to make sure your dependencies are compatible, secure, up-to-date, etc.
- Cons: Rarely updated more than once per quarter; may contain more/less packages than you need for your use case.
- Verdict: If it fits your use case and you can afford the costs, this is the most painless way to deal with dependencies.
Tools For Managing Python Dependency Conflicts
Regardless of which of the above methods you choose to help manage dependency compatibility, you’ll also need a way to manage dependency conflicts. The best practice is to make sure you‘re creating a unique virtual environment for each project. Virtual environments ensure the Python packages you’re installing won’t conflict with any of your other projects, or (more commonly) with your system-level resources. Python provides a number of tools for creating virtual environments, including:
Virtualenv — the classic way
- Pros: Tried and trusted.
- Cons: Need to pip install everything you require; the more environments you create, the harder they are to manage.
- Verdict: Still works well, but requires more manual work than pipenv.
Pipenv — the new Python community standard is a single app that combines both virtualenv and pip.
- Pros: Create a virtual environment and populate it with everything you need from a Requirements.txt file. Supports Pipfile.lock, which contains a fully resolved dependency tree for your project ensuring deterministic builds.
- Cons: Currently supports only a single workflow.
- Verdict: Offers the best way to combat Python dependency issues and ensure environment reproducibility.
To learn more about managing Python dependencies, you might want to watch ActiveState’s Managing Dependencies & Runtime Security deminar.
