setup.py: setup(..., extra_path="foobar", ...)

The extra_path keyword that you can pass to setup() when using the distutils for building and installing a package is completely undocumented as far as I know. I have no idea how I found it the first time (about a year ago) but now it was there and was not doing the Right Thing. The only option I found was to look at the source code of the distutils itself and figure it out from there with much trial and error.

What I'm trying to achieve doesn't seem so weird though. I'm keeping some modules under a src/ directory but don't want them installed in the root package but in .../site-packages/hprof/ with that being imported into the root package via .../site-packages/hprof.pth. Rather similar to what Numeric does I've discovered by now.

Anyway, the end of the story is that had to use the following keywords to setup():

package_dir={'': 'src'},
packages=[""],
ext_modules=[Extension("_hotshot", ["_hotshot.c"])],
extra_path="hprof"

All files, including _hotshot.c live under src/ here, and this ends up installed in the system as described above. According to the distutils code the extra_path keyword is a hack specifically made for Numeric maybe that's why it's not too wel documented.

In case you're wondering what the point is, it's all in that _hotshot module you can see above. I'm creating my own version of that and need to be able to manipulate the sys.path such that it is loaded before the system provided one. Hence the reason for me to install into .../site-packages/hprof/ as I can easily move that earlier in the path before importing _hotshot.

Yes, this also means I'm finally getting an updated version of hprof out soon. Only a year after I got paid by Google for it :-). I must say it's quite disturbing discovering that I screwed up the build system completely last time. It seems I got away with it though... I'll still need to do some more testing tomorrow when I'm more awake, but should get a 0.1.1 out soon. Then build a Debian package and create a patch for the _hotshot.c in the trunk of svn.python.org, which --if it gets accepted-- could remove the need for the custom _hotshot I need to ship now and hence get rid of the weird and complicated build system.

We'll see how that ends, I need to do some job hunting in between too however :-(.