Python Pip & Virtualenv

Using Python virtual environments with pip is the recommended way to install Python packages on Stat servers. Pip is a package manager for Python, capable of installing the latest versions of Python modules. Virtualenv allows you to create multiple Python environments in your home directory.

Together, pip and virtualenv enable you to install Python packages without root (sudo) access, install newer versions of packages than those available by default on a server, and maintain different sets of Python packages for each of your projects.

Usage Example

In this example, a virtualenv named myproject will be created, and the package csv_splitter will be installed.

• If you're on a cluster node, these steps will need to be done from a Slurm interactive session on the build partition, and using the Python Environment Module
• Create a new virtualenv: python3 -m venv myproject
• Activate the virtualenv in your shell session: source ./myproject/bin/activate
  • The virtualenv will always need to be activated before it can be used. For a cluster job, this needs to be included in your sbatch file after the module load Python line.
• Install the latest version of pip in your virtualenv: python3 -m pip install --upgrade pip. If pip is not upgraded first, you may encounter errors trying to install packages on servers with older operating systems.
• Find the name of the package to install: pip search csv
• Install the package: pip install csv_splitter
  • If you encounter errors due to missing system libraries, send an email to help@stat.washington.edu with the error.
• When running Python, you should now be able to load the packages installed via pip
• When you're finished, run the deactivate command to leave your virtualenv

Exporting & Importing Package Lists From Pip

To move your project between computers or servers, you'll need to rebuild your virtualenv to work with the system libraries and Python version on the new computer.

• Export Package List
  • On the old system, activate your virtualenv.
  • Export the list of installed pip packages to a text file: pip freeze > requirements.txt
  • Copy requirements.txt to the new server
• Import Package List
  • On the new system, create a new virtualenv and activate it
  • Install the list of packages from your text file: pip install -r requirements.txt

Further Reading

This is just a quick overview of the basic steps for installing packages with pip and virtualenv. Reading the full documentation is recommended:

• Pip User Guide
• Virtualenv User Guide