E pur si muove

Warnings revisited (aka logging vs warnings)

Wednesday, November 22, 2006

Earlier I posted about how to make warning messages from warnings.warn() look like fairly normal messages so they don't give you all the details about which line of code called them. The result was neat, but I found it a fairly crude way of doing it.

There is another module that does something very similar for getting messages to the user, and even slightly more flexible, and it is called logging. Don't be deceived by it's name like I was, it can be used to just "log" onto the console too. It's even better as the default behaviour is very close to what a console application would need. Let me show the code to get this going:

import logging logging.basicConfig(format='%(levelname)s: %(message)s') logging.warn("Foo is warnging you about bar")

That's it, and it is a lot cleaner then doing this using the warnings module.

Leaves the question, which module to chose. I was wondering too, and a search on "warnings vs logging" only gives one real result. Luckily the explenation of Kristian Reis agrees with what my gut feeling has started to suppose:

warnings are for code `issues', log messages are for application level `issues', right?

I didn't find anyone answering that question, but at least I agree with it. So I reckon it is right...

Overriding formatting of warnings.warn()

Tuesday, November 14, 2006

Today I needed to add the printing of warning messages to a command line tool written in Python. So Instead of writing things out using print, wich is not flexible enough and goes against my way to avoid user interaction deep inside functions and classes, I decided to use the warnings module.

That was easy enough, just send a UserWarning, but I didn't like the way how the warning was printed, with line number and everything. So I had to find a way to make this nicer in this case. What I ended up with is this code in the beginning of the module:

# Overide the default format for warnings.
_formatwarningBackup = warnings.formatwarning
def _formatwarning(message, category, filename, lineno):
if category is UserWarning:
return 'WARNING: %s\n' % message
return _formatwarningBackup(message, category, filename, lineno)
warnings.formatwarning = _formatwarning

It seems a little bit dirty, but I think it is an acceptable solution. If a module somwhere uses the wanrings mechanism for something it should still appear as normal, while my own warnings are masked nicely and conforming with the other output of the program.

As I couldn't find any examples of how to do this when searching, I have now provided one. ;-)

[Sorry about the fact that indentation seems to go all wonky in the code block]

Ubuntu & reportbug

Sunday, September 17, 2006

Lately I only appear to be writing about things I'm unhappy with and general moaning. Sorry for that. But here's another one of them:

An important part of Debian is the BTS, it allows users to report bugs as most people will know is obviously very important. Quite a few essays have been written (for example the first hit for a search) about how to write good bugs and how that can be difficult. The point I'm getting to is that many people/organisations have discovered it's a lot better to hand-hold bug writers and help them as much as possible. Debian has done this in the excellent reportbug program. It will collect lots of information about the program you're having trouble with, will show you open bugs against it etc. Perhaps the most important part is that Debian has discovered that the quality of bug reports improve significantly when reportbug is improved.

As the title suggests this rant is about Ubuntu however. Being a Debian derivative I thought they would have learnt this lesson from Debian and support reportbug too. They do indeed, but the way they do isn't amazing. Here the problems:

  • You are not given a list of open bugs.
    This is rather disturbing. I had to launch my web browser and go to, which redirects to Launchpad but that's fine with me, to see if the bug was reported already. I could not find the bugs ordered by package there, so I tried the very useful Debian shortcut: but with no luck. The only way was to use their search facility. That actually lacks quite a bit, in fact I think every bug I ever reported in Ubuntu has always turned out to be a duplicate, in Debian this ratio has been a lot better.
  • The bugreport is sent to a members-only email list.
    That's right, I got a message back that my email was waiting for moderator approval. Eh? This does not appear to be the case when reporting bugs via launchpad (at least when I used that last, as you might guess I prefer reportbug), so why are reportbug users supposed to be inferior? In fact I don't see the point of this at all. Duplicates aren't filtered and it seems like someone else has to go through the trouble of filing the bug again (I suppose in launchpad). Not very efficient.

So why is this? Why doesn't Ubuntu integrate reportbug better? Why don't they use the Debian experience in this? Why create extra work load? Why can't reportbug use launchpad directly? Sorry, but but this just seems like counter productive

mawk considered harmful

Tuesday, September 12, 2006

Consider the following little awk script:

/^--- [-[:alnum:].\/]+$/ { if ($2 !~ /[-[:alnum:].]+\/debian\//) print $2 }

According the the O'Reilly's sed & awk book it is, as far as my knowledge goes, normal valid awk. It's intended to be used like such:

~$ zcat debian_package_version.diff.gz | awk -f script.awk

The purpose is to list any files that are in the diff which do not live in the package/debian/ path.

I quickly created and tested it, then took it into production in my chroot. Interestingly enough I was pretty sure I had just tested it on this very same file which I'm now using it on, only now it didn't output anything! Double check the code, it still looks all right. Hrm. Then try this:

(chroot)~$ ls -l /usr/bin/awk
lrwxrwxrwx 1 root root 21 Jan 11 2006 /usr/bin/awk -> /etc/alternatives/awk
(chroot)~$ ls -l /etc/alternatives/awk
lrwxrwxrwx 1 root root 13 Sep 12 00:49 /etc/alternatives/awk -> /usr/bin/gawk
(chroot)~$ exit
~$ ls -l /etc/alternatives/awk
lrwxrwxrwx 1 root root 13 Jun 1 16:47 /etc/alternatives/awk -> /usr/bin/gawk

Yes indeed, after installing gawk inside the chroot it worked as expected. I'm really concerned about mawk not doing what I expected as by default, on a Debian system, only mawk is installed (because it is a lot smaller, claims to be faster too). I can barely believe the script above is not portable but would love if someone could point out to me what I did wrong!

Update: Upon closer examination it seems mawk doesn't like the [:alnum:] bit. It claims to adhere to POSIX tho. It would also be nice if it failed somehow. As I understand it the last closing bracket of [[:alnum:]] should create an error if it doesn't understand POSIX character classes.

No awk implementations where hurt in the creation of this post.

site, eggs, etc.

Thursday, September 07, 2006

Before I complained about why .pth files are not processed in files on in your $PYTHONPATH. I got a few pointers in that regard, thanks for them. And now I know why it doesn't happen: you can import foo, bar;; bar.harm() inside a .pth file. Since this happens also when system programs are executed a user can effectively do anything they want, just imagine a python script setuid root. One could argue a script like that should start using #!/usr/bin/env python -S but still.

So the solution is described in the easy_install documentation. Create a $PREFIX/lib/python2.X/site-packages/ with the malicious import statement in forcing the user's path to be processed for .pth files. Ok, that introduces the above mentioned security hole. You can also do it entirely safe (and while only being a user) and create a virtual python in your home directory, which is as I understand it a symlink farm to the system python. A bit a crude hack I think but it works I guess.

Having that out of the way I experimented a bit with the setuptools themselves (while attempting to install TurboGears, which I eventually succeeded in). So here are my initial remarks about it:

  • What an idea to name the frontend easy_install! I didn't really find it too easy either... altho that may be related to the fact that I don't like magic unless I fully understand the said magic.
  • It's still very much in development, options change all the time and some eggs need more recent versions then what I have on my system.
  • Using the bootstrap module, as one egg claims my setuptools is too old, it starts to download a new Python! I've got that installed thanks. (oh yeah, it also failed trying to compile python, but that's almost understandable.)
  • My view is that distutils and setuptools try to replace make and autoconf/automake. So why not follow their lessons? Why do you need an explicit EGG-INFO available on the system? Can't you test it? That would be much more backwards compatible.
  • Also missing from make: uninstall. You can expect to be dropped in /usr/local or ~ so you need to make sure you can delete your own files there again.
  • There's no way, other then unzipping and checking EGG-INFO by hand, to figure out the dependencies.
  • Lastly my killer bug: we live in this modern day and age where we don't trust anyone, yet there's no integrated gpg signature checking! It just downloads stuff from PiPy (or wherever you've pointed -f at)! Yes, I know it only does this during usage of easy_install and not at normal runtime, but that's not an excuse!

I feel like I've ranted too much already and realise people have been working hard on this. While I'm not too enthusiastic I can see some of the merits. In a later post I hope to outline some ideas I've had to solve some of my problems. In an ideal world I'd do some coding too, but I know I don't have lots of time. From what I've been thinking up so far most is possible with the current setuptools infrastructure, which is good as they are being adapted by many people. Only gpg signing will need people to learn a new habit, ideally will ask for this at some time in the future.

Ubuntu & upstart

Friday, September 01, 2006

Upstart is a new sysvinit replacement that Ubuntu wants to use. Scott James Remnant explained before it is innovating in that it is events based. Now he writes more about it, and one quote is this:

So what are events and where do they come from? (Note that this part is under development, so may change in later releases).

What? So this is the fundamental basic idea where everything is build around and you're not yet sure how it works? Scary.

Not that I'm against it all (but I also don't think it'll be the best solution for everything), it fairly intresting and I'm sure there is lots to be learned before you get it perfected. But I find it a bit ambitious that they want to use it by default in Edgy, the next Ubuntu release.

Nuking your home directory

Friday, September 01, 2006
/path/to/some/weird/directory$ rm -rf ~

Yep, I've done that today. It sucks. 1.5 seconds after I did that I hit the power button in the hope the kernel wouldn't have flushed it's buffers yet (hm, that old trick might not work with journaling filesystems). Boot up in single user mode and see half of it is gone. Try using recover but for some bizare reason it can't find any inode within the specified time. I think it sucks and I think I should learn to use debugfs directly.

So syncing back from my other machine (.ssh was gone but I luckily still had .gnupg and a few of the other important ones) I try and recover as much as I can. 3 hours later My machine looks nearly the same again. Only major thing I I'm still suffering from for now is having lost is my .gnome directory, and that contains damn well a lot, but I'm sure more will surface. Most prominent currently are my bookmarks --for once that they where organised a little bit-- and my photo collection.

Originally I was going to write a bit more about the site module and about eggs & setuptools etc. However I've now spent all evening recovering and have also lost the notes about the post. Altho I admit it was 100% user error, I'm still going to blame easy_install for distracting me and giving me this unwanted directory named ~.


When essential tools break...

Saturday, August 26, 2006

I hate it when that happens. Current case:

flub@coronation:/tmp/sandbox$ gpg --clearsign test
gpg: selecting openpgp failed: unknown command
gpg: signing failed: general error
gpg: test: clearsign failed: general error

That's just to get desparate. Some quick poking around results in leading me to think that this only happens when the smart card is involved. --symetric and --list-keys etc seem to work fine.

What's changed on my system? The hardware is still fine as it works on my Debian Sarge box. grrr...

Update: it ends up that seahorse-agent is the bad guy! Disable the agent and all is fine again. I should figure out the details sometime later and amke sure it works. Next problem however is making debsign work with the card nicely, weird that no DD has done that yet.

Update2: seems the debsign problem was just that I was signing from inside my chroot which didn't have access to the required hardware. Oh well, it was late last night.

UTF-8 and Emacs

Thursday, August 24, 2006

While looking around in Manoj's Debian web-page/directory for his new Python policy (for how to package Python stuff for Debian) I stumbled, amongst other intresting things, across this cool UTF-8 demo file.

To my surprise my web browser (Epiphany) shows nearly everything perfect, so does gedit. However my much beloved and favourite religion Emacs, where I spend most of my text-editing time, does not! Even after I installed all the xfonts-intl-* on my Debian box. Surely Manoj knows which packages are needed (given that he even reads mail with Emacs while I use merely mutt), why not write that down into the same file?

No editors or religions where harmed in the creation of this post, a web browser and keyboard have been harmed however as I had to write it into a silly web form wich makes me angry from time to time for not being enough Emacs like.

[I know I should get a better blogging system, maybe ikiwiki is the way to go...]

site module musings

Tuesday, August 22, 2006

The site package reads .pth files and inserts directories listed there in sys.path. However it only scans lib directories (lib/python2.4/site-packages/ and lib/python/site-python/ on UNIX) starting with sys.prefix or sys.exec_prefix for these .pth files. Thus if you install a module distribution using the .pth mechanism but passing --prefix=/foo or --home=/bar to you're in trouble.

Is it such a weird idea to at least scan the directories given in $PYTHONPATH for .pth files? I would even suggest that site just scans the entire sys.path for .pth files, but it might be that I'm overlooking a security issue there (but having an insecure direcotry on sys.path is dangerous anyway and should not happen afaik). I can agree that scanning for a .pth file inside a directory added by a .pth file is rather useless though.

So this leaves me with the idea of filing a bug to change site to scan $PYTHONPATH directories too... If no one stops me I might do that soon.

Python web frameworks

Tuesday, August 22, 2006

It appears discussing web frameworks is popular currently. It also appears GvR has decided he likes Django and hopes it will become sort of the default.

I've been looking at this Django and TurboGears things for a while unable to decide if I should move the website that I appear to be maintaining (which was originally written in PHP and is rather unmaintainable by now) to one of them. I'm not going to add my own comparison here, but I remember from last time I looked at them they both had points I liked and I disliked and I was left undecided. Point is this "blessing" of Django doesn't help me at all. Firstly it's only the personal decision of one person, it doesn't help that that person doesn't like XML so effectively didn't really consider TurboGears.

Today however I had the idea to see if any of the two had any pre-build packages that one can use, like PHP has for forums, photo galleries etc.[1]

I couldn't find a link on the TurboGears website, but using Google I immediately ended up at a part of that was exactly this: a collection of useful applications/widgets. It's called cogbin. I'm not quite sure how it got hidden so well on their site, maybe because there aren't too many available yet. But it's important to note that they integrated it with the Python Cheese Shop by using the trove classifiers. I can't explain how good I find that.

For Django I found nothing of like this. Not on their site and not using Google. Did I just miss it or are they just not as organised?

So maybe I'm tending towards TurboGears for the moment. Their approach also seems more modular which seems to fit more in with the UNIX philosophy in my oppinion. They're even talking about things like markup, SQLAlchemy and Routes. All very exciting things IMHO.

[1] Now we're at it it's maybe also a good time to define interfaces to interact between components like this. I'm thinking mainly about user authentication and creation (but I'm sure there is more). That's a different story though. setup(..., extra_path="foobar", ...)

Monday, August 21, 2006

The extra_path keyword that you can pass to setup() when using the distutils for building and installing a package is completely undocumented as far as I know. I have no idea how I found it the first time (about a year ago) but now it was there and was not doing the Right Thing. The only option I found was to look at the source code of the distutils itself and figure it out from there with much trial and error.

What I'm trying to achieve doesn't seem so weird though. I'm keeping some modules under a src/ directory but don't want them installed in the ``root'' package but in .../site-packages/hprof/ with that being imported into the ``root'' package via .../site-packages/hprof.pth. Rather similar to what Numeric does I've discovered by now.

Anyway, the end of the story is that had to use the following keywords to setup():

package_dir={'': 'src'},
ext_modules=[Extension("_hotshot", ["_hotshot.c"])],

All files, including _hotshot.c live under src/ here, and this ends up installed in the system as described above. According to the distutils code the extra_path keyword is a hack specifically made for Numeric maybe that's why it's not too wel documented.

In case you're wondering what the point is, it's all in that _hotshot module you can see above. I'm creating my own version of that and need to be able to manipulate the sys.path such that it is loaded before the system provided one. Hence the reason for me to install into .../site-packages/hprof/ as I can easily move that earlier in the path before importing _hotshot.

Yes, this also means I'm finally getting an updated version of hprof out soon. Only a year after I got paid by Google for it :-). I must say it's quite disturbing discovering that I screwed up the build system completely last time. It seems I got away with it though... I'll still need to do some more testing tomorrow when I'm more awake, but should get a 0.1.1 out soon. Then build a Debian package and create a patch for the _hotshot.c in the trunk of, which --if it gets accepted-- could remove the need for the custom _hotshot I need to ship now and hence get rid of the weird and complicated build system.

We'll see how that ends, I need to do some job hunting in between too however :-(.

Today's WTF: sys.path in python != sys.path in ipython

Thursday, August 17, 2006

Spot the current directory (<snip>/build/lib.linux-i686-2.4) in the path:

<snip>/build/lib.linux-i686-2.4$ echo $PYTHONPATH

<snip>/build/lib.linux-i686-2.4$ python
Python 2.4.3 (#2, Apr 27 2006, 14:43:58)
[GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '<snip>/build/lib.linux-i686-2.4', '/home/flub/lib/python', '/home/flub/lib/python2.4', '/usr/lib/', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/lib/python2.4/site-packages', '/usr/lib/site-python']
>>> Ctrl-D

<snip>/build/lib.linux-i686-2.4/$ ipython
In [1]:import sys
In [2]:sys.path
In [3]: Ctrl-D

This seriously screws up my interactive sessions with ipython since the module in /usr/lib gets loaded before my own modified one. Why does ipython think it is a good idea to do it that way?

Since we're at the subject, notice the PYTHONPATH environment variable. Why does just the existence of this variable -- what it is set to does actually not matter at all -- make the current directory show up in the path. Also what's the point of the empty ('') string at the beginning of the path? What does that do?

Lots of questions, I know, but I haven't found any of the answers anywhere so far. Maybe I have just looked to hard, but I'm very intrigued.

GnuPG Smartcard on Debian Sarge

Wednesday, June 21, 2006

There's a document on the GnuPG explaining how to set up your system to use a smartcard. There a few people around claiming that it works on a standard Debian Sarge by using udev. Up to today I have had no luck with that however, but that finally changed!

I'll skip narrating all the thing I tried in vain, including compiling new kernels from So if you happen to have an SCM Microsystems SCR335 smartcard reader (which is listed on the HOWTO as supported) like me this should get it to work.

Packages you'll have to install:

  • gnupg
  • pcscd
  • libpcsclite-dev
  • a 2.6 linux kernel
  • udev

Everything else will come in as a dependency.

After that you also need a file in /etc/udev/rules.d/gnnupg-ccid.rules and no, don't use the one they suggest in the HOWTO. This one line is much better as it doesn't need the speparate script (and actually works!):

BUS=="usb", ENV{ACTION}=="add", SYSFS{idVendor}=="04e6", SYSFS{idProduct}=="5115", GROUP="users", MODE="0660"

Note the ENV bit. You'll have to adjust the SYSFS values if you have a different reader. Lastly there's the GROUP and MODE to play with, I really don't see why you need to create yet another group for this. Why wouldn't all users be allowed to use a smartcard?

Now you should have the green led on the reader on when you insert the card. And gpg --card-status should give you output too. Start playing.

If you find any errors in the above (I missed a required package or so) please tell me and I'll update the description.

Dalai Lama visits Belgium

Friday, June 02, 2006

Like I care.

But China cares, that's more worrying. Dispite it not being an "official" visit (he was invited by some buddhists to open a new temple) he visited the prime minister and some other politicians. And hence the chinese ambassador says it will hurt the Belgian-China relationships as they don't like him. "The territorial integrity of a country are very important and is in the heart of every chinese." Yeah right.

The final quote is the best: "I hope that not only we, but also the Belgian government and politicians, will realise that the Belgian-China relationship is currently more important." In other words: screw human rights, it's good for our economy.

The worst part of it all is that our politicians try and not offend China too much. I've already a few time wondered if it is possible to boycot 'made in china' stuff. But that seems so damn difficult.

Summer of Code 2006

Friday, May 26, 2006

There must be about a million blog posts with this title by now. Anyway, here my own little addition, I've been meaning to write this for a while.

I have not applied for SoC again althoug I was still eligible. There where intresting projects however, and in fact I'm sorry to see now that the list of accepted projects is known that some of the ones I was intrested in are not being done (e.g. the only accepted bazaar one is bazaar-gui afaik, and I was looking at bazaar).

There are a few reasons I decided not to apply. Firstly the timeline has been extended, but mainly moved forward which did not fit very well with my university shedule. Secondly I felt that I had a great opportunity last year, and wanted to give this opportunity to more new people.

Then there is also the aspect of finally graduating this year (hopefully!). It means I should really try and find a job instead of postponing that a bit longer. And not having the looming SoC deadline means I can take a more exciting holiday climbing on the outer hebrides on some silly islands where no one lives and with no contact with the rest of the world. We did that 2 years ago and I'm really looking forward to spending July there.

Lastly not having the SoC project eating most of my time means I can take time to complete various personal projects, including my SoC 2005 project. I'd really like to polish it off completely and finally make a proper Debian package for it. Maybe it could even be on time for the etch release! The other Debian package I'm maintaining, plotutils, also needs some of my attention which it hasn't been getting due to my over-busy uni work lately.

So that's about my plan for this summer. Finish off lasts year project and other duties I took upon me, and trying to find work to do.

Subscribe to: Posts (Atom)