Sebastian Rittau's Blog

Thursday
Jul 26 2007

setuptools breaks PYTHONPATH

Python, Programming, setuptools, PYTHONPATH

setuptools and egg files are a great way to distribute Python packages and programs. But today I stumbled over a really braindead design decision: setuptools overrides Python's standard module search path algorithm in a very inconvenient way. Normally, when Python looks for a module or package, it first looks in the current directory, then in any path specified in the PYTHONPATH environment variable, and finally in the default search path. setuptools changes the lookup order to the following:

  1. the current directory
  2. egg files specified in PYTHONPATH
  3. egg files in the default search path
  4. regular modules in PYTHONPATH
  5. regular modules in the default search path

This means that an egg file in the default search path overrides a non-egg module in PYTHONPATH. I wonder what the rationale for that decision is (and it seems to be a deliberate decision). PYTHONPATH is a means to override the default search path and is perfectly suited for testing, developing, and forcing to use a particular (e.g. a bundled) version of a package. Not anymore! Now, the only way to override an egg is to create your own egg. And this is non-trivial and often means an extra step during development.

Here is where it bites me: I have created a library of custom classes, that I use in several projects. I break API frequently, since it is developed in a just-in-time fashion: whenever I need some functionality, I implement it. Therefore I bundle the library (or parts of it) with most projects. This works well for projects that go into a separate directory, like web projects. Nevertheless I use the library for some other projects that are installed system-globally. Otherwise I would need some way to namespace it for the different projects. Not my idea of fun. Normally, this is not a problem, since I can just set PYTHONPATH for all projects that have a local copy, at least when developing. Due to the strange path handling of setuptools, this method breaks and I have to work around it. Great ...

Comments

easy_install -m

by Phillip J. Eby

There's a trivial fix: install eggs in "multi-version" mode, the -m option to easy_install. When you do that, NO eggs will be on sys.path at all, unless you specifically ask for them.

You can do this for just one egg, or do it for everything you have installed.

If you don't use -m, easy_install assumes you want the things you install to be THE package to use, period.