Bundling Up Distributable Python Package Libraries Using pip and ZIP

In order to utilize some of the awesome available Python libraries with Google App Engine, the libraries are best packaged into a ZIP package and distributed alongside the other files under development. It’s kind of like JARing stuff up in Java, it makes it so that you never have to mess with the library packages, and you can have one stable library base to work with, until you decide to update it all again.

I haven’t been able to find a useful guide to do this via the following searches (which I’m including so Google might pick up on them):

“bundling multiple python modules”
“bundling python packages into zip”
“using pip to create zip packages”
“building library package pip”

The standard pip installer program, which is used to distribute and install so many Python packages, must have a way of doing this.

First, let’s see what is currently installed in the virtualenv that I’m using to do my development. In this case, I’m using Flask and a number of items related to it:

Now, how do we bundle these up?

It looks something like this (copied from my project’s requirements.txt):

First, you want to make sure to create a new virtualenv and to activate it before using the Bash script. If you then run pip freeze in the activated packageenv, you should see a minimal number of packages, possibly none at all. In my case, VirtualBox pushed its packages into the global Python site-packages folder.

Then, inside of the clean virtualenv, the Bash script (packages.sh) to run would look like this (on OS X):

When the script is finished, the packages.zip would look something like:

The packages.zip file will probably contain the test suites of the included libraries as well. I haven’t made any provisions to delete these files in the Bash script.

Now, in the Python file that wants access to these libraries, you insert the packages.zip bundle early enough into the sys.path variable to make Python use it to dereference imports. That said and done, you now have access to all the packages necessary to run the full Flask WSGI-compliant server + its plugins + whatever plugins you want to bundle. This all helps to keep your dev environment clean and it helps that you can just re-run the Bash script to update the entire bundle.

For example (packages.py):

You could now actually use a clean virtualenv with this packages.zip bundle to do your development, and consolidate all of the development into a single standalone directory, with no dependencies on the user or global site-packages.