Managing Python versions with Pyenv
There are several ways to install Python on your system and each come with their advantages and disadvantages. For example, a data scientist will benefit from the pre-installed packages like SciPy and Numpy from a quick Anaconda installation. For developers however, it’s better to use tools that provide ease of switching between different versions of Python as your projects may require specific versions. These tools are:
- Pyenv – Python version management tool to manage multiple Python installations
- Poetry – Manages Python virtual environments and the project dependencies within
If you’re on Mac you may have a default Python 2.7 aleady installed on your system (although Apple has finally come to removing it from macOS 12.3). If you have also installed other Python versions through Anaconda or from python.org, you’ll never quite know for sure which Python interpreter is being used when you run commands in your terminal.
Pyenv solves this problem by managing your Python executables in a simple and elegant way and here’s how to install it.
How does Pyenv work? Pyenv inserts $HOME/.pyenv/shims at the beginning of PATH to intercept calls to Python. A ‘shim’ is a special executable that will redirect the call to a specific python version. There is an order to which the python version will be selected, depending on the configuration: Shell: The environment variable PYENV_VERSION. This can be set with the command export PYENV_VERSION=<python_version> Local: This is set in a .python_version file which is generated using the command pyenv local <python_version>. It will be searched recrusively from the current directory until it reaches the root directory. Global: This can be generated with command pyenv global <python_version> . System: When there is no configuration found, the python version will resort to a default one already installed on the operating system.
Installing Pyenv
Windows users can follow the instructions in the Github repo here
Mac (using Homebrew):
Once installed, we can first see a list of all python executables, if they are already installed on your system.
Output:
I already have multiple python versions installed on my system and the global python interpreter has been set to 3.8.10 as shown by the asterisk. This can be checked by running
Let’s now list all the available python versions available to install
Output:
There are many to choose from, but we’ll assume we have no python versions installed yet and choose 3.8.11 to install no
Once installed, if we then run pyenv versions again, we’ll see our new version in the list. It won’t yet be active and to avoid any conflicts with default python installations you may already have, we will not change the global python interpreter and instead we will allocate our python versions per project.
Managing dependencies with Poetry
It’s best practice to install dependencies per project to isolate the specific versions of each package and to prevent conflicts. Virtual environments are the best solution and we’ll let Poetry manage this for us.
Installing Poetry
Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and will manage them for you. To install Poetry run the following command
Mac
Windows Powershell
To apply the installation to your current shell session, run
It can also be added to .bashrc or .zshrc files
Creating a new Poetry project
Poetry will create a new folder structure for you, so in your parent project directory ru
📦my_new_project
┣ 📂my_new_project
┃ ┗ 📜__init__.py
┣ 📂tests
┃ ┣ 📜__init__.py
┃ ┗ 📜test_my_new_project.py
┣ 📜README.rst
┗ 📜pyproject.toml
For this project, let’s allocate the python version 3.8.11 we already downloaded before with Pyenv. We can use the ‘local’ command while in the root direcotry of the new project
Running ‘pyenv versions’ from this location will now show Python 3.8.11 as the Python version marked with an asterisk to indicate it is active for this directory.
Our next task is to tell Poetry to use this new python we just allocated to this project directory. To do this, run
Output:
As shown in the output above, the virtual environments by default are store in ../Caches/pypoetry/virtualenvs. It may be better that each virtual environment is created inside the project root instead as this will allow editors like VSCode to automatically load the virtual environment. To change the configuration for poetry to do this, run
For more information on configuring Poetry, visit the configuration section in the documentation.
Poetry comes with commands to add or remove packages, update packages and more so you don’t need to manually configure any files. In the snippet below, we will add the package FastAPI
The pyproject.toml file will be updated automatically with information about dependencies. If you wish to install development packages, you can use the option –dev when adding the package, and these will be installed to a separate section in the pyproject.toml file
This covers the basics of setting up a new project with Pyenv and using Poetry to install your dependencies. For more information about the tools, check the websites linked at the top of this page for documentation.
Source: https://moderndataengineering.substack.com/