Category: Python

Bundling Up Distributable Python Package Libraries Using pip and ZIP

In order to utilize some of the awesome available Python libraries with Google App Engine, the libraries are best packaged into a ZIP package and distributed alongside the other files under development. It’s kind of like JARing stuff up in Java, it makes it so that you never have to mess with the library packages, and you can have one stable library base to work with, until you decide to update it all again.

I haven’t been able to find a useful guide to do this via the following searches (which I’m including so Google might pick up on them):

“bundling multiple python modules”
“bundling python packages into zip”
“using pip to create zip packages”
“building library package pip”

The standard pip installer program, which is used to distribute and install so many Python packages, must have a way of doing this.

First, let’s see what is currently installed in the virtualenv that I’m using to do my development. In this case, I’m using Flask and a number of items related to it:

Now, how do we bundle these up?

It looks something like this (copied from my project’s requirements.txt):

First, you want to make sure to create a new virtualenv and to activate it before using the Bash script. If you then run pip freeze in the activated packageenv, you should see a minimal number of packages, possibly none at all. In my case, VirtualBox pushed its packages into the global Python site-packages folder.

Then, inside of the clean virtualenv, the Bash script (packages.sh) to run would look like this (on OS X):

When the script is finished, the packages.zip would look something like:

The packages.zip file will probably contain the test suites of the included libraries as well. I haven’t made any provisions to delete these files in the Bash script.

Now, in the Python file that wants access to these libraries, you insert the packages.zip bundle early enough into the sys.path variable to make Python use it to dereference imports. That said and done, you now have access to all the packages necessary to run the full Flask WSGI-compliant server + its plugins + whatever plugins you want to bundle. This all helps to keep your dev environment clean and it helps that you can just re-run the Bash script to update the entire bundle.

For example (packages.py):

You could now actually use a clean virtualenv with this packages.zip bundle to do your development, and consolidate all of the development into a single standalone directory, with no dependencies on the user or global site-packages.

Serving Static Files From Root “/” and not “/static” Using Flask

Note to self.

There’s some discussion about how to do this here, involving Werkzeug’s middleware.
http://stackoverflow.com/questions/4239825/static-files-in-flask-robot-txt-sitemap-xml-mod-wsgi

This is another way to do it, assuming you have a folder called “static” under where the main application .py file is located:

from flask import Flask, request
app = Flask(__name__, static_url_path='')

@app.route('/')
def root():
return app.send_static_file('index.html')

By setting static_url_path to a blank string, it means any accesses to unrouted root URLs, will attempt to grab the associated file out of the static folder. What it also means is that if you have HTML in that folder (as would be the case if you’re using Node or some other server), the hrefs do not look like “/static/js/something.js” but just “/js/something.js”, which may be useful.

Python virtualenv error when calling easy_install

If you see the following exception when trying to set up a Python virtualenv environment on Mac OS X (or possibly other OSes), the culprit is usually the contents of .pydistutils.cfg.

$ virtualenv flaskenv
New python executable in flaskenv/bin/python
Please make sure you remove any previous custom paths from your /Users/account/.pydistutils.cfg file.
Installing setuptools.............done.
Installing pip....
Complete output from command /Users/account/devsrc/flaskenv/bin/python -x /Users/account/devs...env/bin/easy_install /Users/account/Libr...ort/pip-1.2.1.tar.gz:
/Users/account/devsrc/flaskenv/bin/python: can't open file '/Users/account/devsrc/flaskenv/bin/easy_install': [Errno 2] No such file or directory
----------------------------------------
...Installing pip...done.
Traceback (most recent call last):
File "/Users/account/bin/virtualenv", line 8, in
load_entry_point('virtualenv==1.8.2', 'console_scripts', 'virtualenv')()
File "/Users/account/Library/Python/2.7/site-packages/virtualenv.py", line 942, in main
never_download=options.never_download)
File "/Users/account/Library/Python/2.7/site-packages/virtualenv.py", line 1054, in create_environment
install_pip(py_executable, search_dirs=search_dirs, never_download=never_download)
File "/Users/account/Library/Python/2.7/site-packages/virtualenv.py", line 655, in install_pip
filter_stdout=_filter_setup)
File "/Users/account/Library/Python/2.7/site-packages/virtualenv.py", line 1020, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /Users/account/devsrc/flaskenv/bin/python -x /Users/account/devs...env/bin/easy_install /Users/account/Libr...ort/pip-1.2.1.tar.gz failed with error
code 2

You’ve just gotta comment out the settings in .pydistutils.cfg:

$ cat ~/.pydistutils.cfg
#[install]
#install_lib = ~/Library/Python/$py_version_short/site-packages
#install_scripts = ~/bin

Then everything works as expected:

$ virtualenv flaskenv
New python executable in flaskenv/bin/python
Please make sure you remove any previous custom paths from your /Users/account/.pydistutils.cfg file.
Installing setuptools............done.
Installing pip...............done.

Installing Mercurial Without Admin Rights And No C Compiler

Cheap webhosters can be pretty limiting sometimes in the platforms they run and the software packages on offer, even when you do have shell access. Here’s a quick guide to setting Mercurial up on a shared webhost for use when:

  • there is no Python.h available, so you cannot build the C extensions
  • you can build only the pure Python modules, which is slower but more compatible
  • you have to do all of this in your home directory somewhere

Continue reading Installing Mercurial Without Admin Rights And No C Compiler