devork

Small Emacs tweaks impoving my Python coding

2011-09-15T01:21:00.000+01:00

Today I've spent a few minutes tweaking Emacs a little. The result is very simple yet makes a decent impact on usage.

Firstly I remembered using the c-subword-mode a long time ago, I couldn't believe I never used that in Python. Turns out there is a more genericly named subword-mode by now (emacs 23 IIUC) and it's very easy to enable for Python by default:

(add-hook 'python-mode-hook (lambda () (subword-mode 1)))

The second simple improvement was finally figuring out how to automatically enable flyspell-mode when editing Restructured Text:

(add-hook 'rst-mode-hook (lambda () (flyspell-mode 1)))

Both where so simple I can't believe I didn't do them earlier.

And while on the subject, develock-mode is great and I've been using it a very long time. Unfortunately python isn't supported out of the box, but no worries, someone has done the work already. So all you have to do is something along the lines of:

(load "~/.emacs.d/develock-py.el")

I don't think I'd still want to edit a python file without it

Using getattr and property

2011-06-17T19:25:00.000+01:00

Today I wasted a lot of time trying to figure out why a class using both a __getattr__ and a property mysteriously failed. The short version is: Make sure you don't raise an AttributeError in the property.fget()

The start point of this horrible voyage was a class which looked roughly like this:

class Foo(object):

    def __getattr__(self, name):
        return self._container.get(name, 'some_default')

    @property
    def foo(self):
        val = self._container.get(foo)
        if test(val):
            return some_helper(val)
        return val

This sees fine enough. Only it turns out that some_helper() raised an AttributeError for some invalid input. Certainly reasonable since it was never meant to deal with incorrect input, that was meant to have been sanitised already (that was a bug in the caller which was actually just an incorrect unittest). The main gotcha was that it seems that python doesn't just check whether a "foo" is present in all the relevant __dict__'s along the mro. Instead it seems to use getattr(inst, "foo") and then delegate to __getattr__() if it gets an AttributeError. Now suddenly finding a bug in some_helper() has turned into a puzzling question as to why __getattr__() was called.

Personally I can't see why it doesn't use the mro to statically look up the required object instead of using the AttributeError-swallowing approach. But maybe there's a good reason.

Synchronising eventlets and threads

2011-03-17T19:43:00.002Z

Eventlet is an asynchronous network I/O framework which combines an event loop with greenlet-based coroutines to provide a familiar blocking-like API to the developer. One of the reasons I like eventlet a lot is that the technology it builds on allows it's event loop to run inside a thread or even run multiple event loops in different threads. This makes it a lot more amenable to slowly evolving existing applications to be more asynchronous then solutions like e.g. gevent which only allow one event loop per process.

Eventlet isn't the most mature of tools however and it's API shows signs of being developed as needs arose. Not that this is necessarily a bad thing, APIs do need to grow from being used, but don't be surprised if you need to dig down into some parts and discover rough edges (hi IPv6!). The API does have a decent collection of tools you'll be familiar with however: greenlet-local storage, semaphores, events (though not quite the event you're used too) and even some extra goodies like pools, WSGI servers, DBAPI2 connection pools and ZeroMQ support. But you'll notice that all these goodies are designed to work in a greenlet-only world. And the one place where threads are acknowledged, a global threadpool of workers as a last resort to make things behave asynchronous, looks very messy and entirely not reusable (it's full of module globals for one). So if you're introducing eventlet into an existing application and you need to communicate data and events between threads and eventlets you'll find a void.

So after some studying of the tpool module I decided to build a class which could synchronise between threads and eventlets. This class, which I called a Notifier, can be basically thought of as a Condition without the lock, i.e. there are three methods: .wait(), .notify() and the rather similar .notify_all(). The idea is that any thread or eventlet which calls .wait() will block (cooperatively block in the case of a greenlet) until it gets woken up by a call to one of the notifying methods. That's all there is to it.

Building a Notifier

(Be prepared to look at the source code for eventlet.hubs.hub, threading and related code when reading this.)

Firstly the class will need to be constructed. For now there's only one interesting instance attribute and that is _waiters which is a set which will contain all the threads and eventlets currently blocking on a call to .wait().

def __init__(self, hubcache=GLOBAL_HUBCACHE):
        self._waiters = set()
        self.hubcache = hubcache

Don't worry yet about what goes into the set of waiters and also ignore the hubcache for now. We'll get to those later.

Now lets build the .wait() method: it needs to block until notified. But blocking is significantly different when you're running in a thread then when running in an eventlet. The basics are that in a thread you want to really block using the locking primitives provided by the OS (exposed to Python in the thread module) while an eventlet essentially wants to switch to the event loop, called the hub, in the hope someone will eventually switch back to it when it needs to wake up. These two are so different that they easily divide in two methods: .gwait() for blocking eventlets and .twait() for blocking threads.

def wait(self, timeout=None):
    hub = eventlet.hubs.get_hub()
    if hub.running:
        self.gwait(timeout)
    else:
        self.twait(timeout)

You could call these directly obviously, but as you can see abstracting them away is not that hard. You can easily detect if you're in an eventlet by checking if the hub (which is thread-local) is actually running. The price you pay for this is that this will create a hub instance in each thread, even if it is not used. But the worst this does is waste some memory. (This could be avoided by checking for the eventlet.hubs._threadlocal.hub attribute, but that's even more internal then .get_hub().)

As already mentioned the basics of blocking in an eventlet is to switch to the hub and then wait until some other greenlet running in the same thread switches back to you. So a notifier needs to have a reference to your geenlet instance so it can call .switch() on it when the time comes to wake you up. But what does another thread do? Well it turns out another thread could ask your hub to schedule a function to run in your thread using the hub's .schedule_call_global() method since the only thread-critical operation is an append on the hubs's next_timers list, which is a thread-safe operation. Now there is another catch, remember that the hub is basically an event loop? Well if no events happen then it will not be going round it loop! And appening something to a list is not creating an event. So what you need to do is use os.pipe so you have a filedescriptor which you can register with the hub and now you can just write some data into this pipe when you want the hub of the waiter to wake up. Setting up this pipe is what the mysterious call to ._create_pipe() does, we'll see it in detail later. The rest is just simple sugar: dealing with timeouts and returning the correct values for them:

def gwait(self, timeout=None):
    waiter = eventlet.getcurrent()
    hub = eventlet.hubs.get_hub()
    self._create_pipe(hub)
    self._waiters.add((waiter, hub))
    if timeout and timeout > 0:
        timeout = eventlet.Timeout(timeout)
        try:
            with timeout:
                hub.switch()
        except eventlet.Timeout, t:
            if t is not timeout:
                raise
            self._waiters.discard((waiter, hub))
            return False
        else:
            return True
    else:
        hub.switch()
        return True

Next on lets look at how you block in a thread. This is actually surprisingly simple, just copy what threading.Condition.wait() does for it: it is perfectly non-blocking for a greenlet to call .release() on a lock which was acquired by another thread. Notice the ugly CPU-consuming spinning which happens when a timeout is in use, luckily this has been fixed in python 3.2 (issue7316).

def twait(self, timeout=None):
    waiter = threading.Lock()
    waiter.acquire()
    self._waiters.add((waiter, None))
    if timeout is None:
        waiter.acquire()
        return True
    else:
        # Spin around a little, just like the stdlib does
        _time = time.time
        _sleep = time.sleep
        min = __builtin__.min
        endtime = _time() + timeout
        delay = 0.0005      # 500 us -> initial delay of 1 ms
        while True:
            gotit = waiter.acquire(0)
            if gotit:
                break
            remaining = endtime - _time()
            if remaining <= 0:
                break
            delay = min(delay * 2, remaining, .05)
            _sleep(delay)
        if not gotit:
            self._waiters.discard((waiter, None))
            return False
        else:
            return True

The last thing of interest here is how the hub does not matter, so we just place None in the set of waiters.

Now lets have a look at what notify looks like. We've already discussed what needs to happen here: to notify an eventlet we need schedule a call with it's hub to switch to it (no point special casing when we're already running in that same hub). In case we notified it from a different thread we also need to signal the hub using the pipe set up by .wait() so it will actually start to go round it's loop and execute this just scheduled call. Notifying a thread is even easier: just unlock the lock it's trying to acquire.

def notify(self):
    if self._waiters:
        waiter, hub = self._waiters.pop()
        if hub is None:
            # This is a waiting thread
            try:
                waiter.release()
            except thread.error:
                pass
        else:
            # This is a waiting greenlet
            def notif(waiter):
                waiter.switch()
            hub.schedule_call_global(0, notif, waiter)
            if hub is not eventlet.hubs.get_hub():
                self._kick_hub(hub)

Admittedly I've hidden some of the cute trickery away in those ._create_pipe() and ._kick_hub() calls, so lets leave the boring .notify_all() and skip straight to them.

The principle for this is that each eventlet which calls .gwait() needs to ensure there is a pipe available to which a thread can write something. If the hub of the eventlet was waiting for the reading end of this pipe to become readable it will wake up and notice it has to run the notif() function which was scheduled by our call to .notify(). But creating a pipe for each call to .gwait() does seem rather wasteful and this is where the mysterious hubcache comes into play: it is a dictionary keeping track of the pipe associated with each hub.

def _create_pipe(self, hub):
    if hub in self.hubcache:
        return
    def read_callback(fd):
        os.read(fd, 512)
    rfd, wfd = os.pipe()
    listener = hub.add(eventlet.hubs.hub.READ, rfd, read_callback)
    self.hubcache[hub] = (rfd, wfd, listener)

You can see this asks the hub to wake up when the reading end of the created pipe becomes readable and when this happens read_callback() will be called. The only purpose of read_callback() is to read all the data written to the pipe so that the OS buffers are emptied and the pipe can be re-used to wake the hub up the next time.

Now there is just ._kick_hub() left. This should now be obvious: look up the hub in the hubcache and write some data to the writing end of the pipe. The only gotcha here is that while we might be called from another thread, this does not mean the calling thread itself can't be part of an eventlet mainloop. So in that case make sure not to do a blocking write (as unlikely as that might be).

def _kick_hub(self, hub):
    rfd, wfd, r_listener = self.hubcache[hub]
    current_hub = eventlet.hubs.get_hub()
    if current_hub.running:
        def write(fd):
            os.write(fd, 'A')
            current_hub.remove(w_listener)
        w_listener = current_hub.add(eventlet.hubs.hub.WRITE, wfd, write)
    else:
        os.write(wfd, 'A')

Notice here how in the async case we remove the listener which was used to wake the hub up right from the callback itself. If we didn't do this then the hub would most likely find the writing end of the pipe writable again on the next loop which would trigger another notification of the other hub, not what we want!

That's cute, but what now?

We've now got a great way of notifying other threads and eventlets at will. But this is an entirely non-standard tool! Using this is strange, unfamiliar and unwieldy. This isn't one of the synchronisation primitives we know and wanted to use.

But look how easy it is to build a lock now: we just need a real lock and one of these strange notifiers:

class Lock(object):

    def __init__(self, hubcache=GLOBAL_HUBCACHE):
        self.hubcache = hubcache
        self._notif = Notifier(hubcache)
        self._lock = threading.Lock()
        self.owner = None

    def acquire(self, blocking=True, timeout=None):
        gotit = self._lock.acquire(False)
        if gotit or not blocking:
            if gotit:
                self.owner = eventlet.getcurrent()
            return gotit
        if timeout is None:
            while not gotit:
                self._notif.wait()
                gotit = self._lock.acquire(False)
            if gotit:
                self.owner = eventlet.getcurrent()
                return True
        else:
            if timeout < 0:
                raise RuntimeError('timeout must be greater or equal then 0')
            now = time.time()
            end = now + timeout
            while not gotit and (now < end):
                self._notif.wait(end - now)
                gotit = self._lock.acquire(False)
                now = time.time()
            if gotit:
                self.owner = eventlet.getcurrent()
            return gotit

    __enter__ = acquire

    def release(self):
        self._lock.release()
        self.owner = None
        self._notif.notify()

    def __exit__(self, exc_type, exc_value, traceback):
        self.release()

    def __repr__(self):
        return ('<gsync.Lock object at 0x%x (%r, %r)>' %
                (id(self), self.owner, self._notif))

Most of this code is to deal with the timeout of the lock. If you would only implement a lock as how it was before Python 3.2 this would have been even simpler. (Oh, also notice the owner attribute, that will come in handy for the next tool.)

Of course now we have a lock we can easily build the next tool: a proper Condition. It's almost like I planned this!

class Condition(object):

    def __init__(self, lock=None, hubcache=GLOBAL_HUBCACHE):
        if lock is None:
            self._lock = Lock(hubcache=hubcache)
        else:
            self._lock = lock
        self._notif = Notifier(hubcache=hubcache)

        # Export the lock methods
        self.acquire = self._lock.acquire
        self.release = self._lock.release
        self.__enter__ = self._lock.__enter__
        self.__exit__ = self._lock.__exit__

    def __repr__(self):
        return '<gsync.Condition (%r, %r)>' % (self._lock, self._notif)

    def wait(self, timeout=None):
        if self._lock.owner is not eventlet.getcurrent():
            raise RuntimeError('Can not wait on un-acquired lock')
        self._lock.release()
        try:
            self._notif.wait(timeout)
        finally:
            self._lock.acquire()

    def notify(self):
        if self._lock.owner is not eventlet.getcurrent():
            raise RuntimeError('Can not notify on un-acquired lock')
        self._notif.notify()

    def notify_all(self):
        if self._lock.owner is not eventlet.getcurrent():
            raise RuntimeError('Can not notify on un-acquired lock')
        self._notif.notify_all()

No surprises here. This is truly getting trivial to implement thanks to our previous two primitives.

Now once we have a condition we can finally get to the real prise: a queue to move data freely between threads and eventlets.

import Queue as queue

class BaseQueue(object):

    def __init__(self, maxsize=0, hubcache=GLOBAL_HUBCACHE):
        self.hubcache = hubcache
        self.maxsize = maxsize
        self._init(maxsize)
        self.mutex = Lock(hubcache=hubcache)
        self.not_empty = Condition(self.mutex, hubcache=hubcache)
        self.not_full = Condition(self.mutex, hubcache=hubcache)
        self.all_tasks_done = Condition(self.mutex, hubcache=hubcache)
        self.unfinished_tasks = 0

class Queue(BaseQueue, queue.Queue):
    pass

Great, only had to provide a new .__init__() which uses our own locking primitives, everything else of the stdlib Queue class can be re-used.

But why the strange diversion to create a separate BaseQueue class? Well, it makes making the priority and lifo queues very easy:

class PriorityQueue(BaseQueue, queue.PriorityQueue):
    pass

class LifoQueue(BaseQueue, queue.LifoQueue): pass

That's right, mro FTW!

Caveats when mixing threads and eventlets

There is one issue to watch out for: imagine a thread which consumes items from a queue, spawning eventlets to do the work.

class Worker(threading.Thread):
    def __init__(self, inputq):
        self.inputq = inputq

    def run(self):
        eventlet.sleep(0)  # start the hub
        while True:
            item = self.inputq.get()
            if item is PoisonPill:
                break
            eventlet.spawn(self.do_stuff, item)

Notice that eventlet.sleep(0) line? It basically switches to the hub, thereby implicitly starting it, which then switches back immediately. But why?

Remember the code for Notifier.wait(), it checked if the hub was running to detect whether it was being called from inside an eventlet or not. So if you didn't manage to start the hub before calling this method the whole thread will block! Hence the minor hack to start the hub manually beforehand.

All the code

Here all the code for the notifier in one piece, including the docstrings and comments. This also includes the global hubcache with a tiny bit of extra magic to be able to clear the cache if you so desire. Having this as a parameter to pass in allows you to use different hub caches if you have a reason to do so.

import os
import thread
import threading
import time

import eventlet


class HubCache(dict):
    """Cache used by Notifier instances

    This is a dict-subclass to overwrite the .clear() method.  It's
    keys are hubs and values are (rfd, wfd, listener).  Using this
    means you can clear the cache in a way which will unregister the
    listeners from the hubs and close all filedescriptors.

    XXX This is hugely incomplete, only remove items from this cache
        using the .clear() method as the other ways of removing items
        will not release resources properly.
    """

    def clear(self):
        while self:
            hub, (rfd, wfd, listener) = self.popitem()
            hub.remove(listener)
            os.close(rfd)
            os.close(wfd)

    def __del__(self):
        self.clear()


"""The global hubcache

This is the default hubcache used by Notifier instances.
"""
GLOBAL_HUBCACHE = HubCache()


class Notifier(object):
    """Notify one or more waiters

    This is essentially a condition without the lock.  It can be used
    to signal between threads and greenlets at will.
    """
    # This doesn't use eventlet.hubs.trampoline since that results in
    # a filedescriptor per waiting greenlet.  Instead each eventlet
    # that calls .gwait() will ensure there's a filedescriptor
    # registered for reading for with it's hub.  This filedescriptor
    # is then only used when another thread wants to wake up the hub
    # in order for a notification to be delivered to the eventlet.

    def __init__(self, hubcache=GLOBAL_HUBCACHE):
        """Initialise the notifier

        The hubcache is a dictionary which will keep pipes used by the
        notifier so that only ever one pipe gets created per hub.  The
        default is to share this hubcache globally so all notifiers
        use the same pipes for intra-hub communication.
        """
        # Each item in this set is a tuple of (waiter, hub).  For an
        # eventlet the waiter is the greenlet while for a thread it is
        # a lock.  For a thread the hub item is always None.
        self._waiters = set()
        self.hubcache = hubcache

    def wait(self, timeout=None):
        """Wait from a thread or eventlet

        This blocks the current thread/eventlet until it gets woken up
        by a call to .notify() or .notify_all().

        This will automatically dispatch to .gwait() or .twait() as
        needed so that the blocking will be cooperative for greenlets.

        Returns True if this thread/eventlet was notified and False
        when a timeout occurred.
        """
        hub = eventlet.hubs.get_hub()
        if hub.running:
            self.gwait(timeout)
        else:
            self.twait(timeout)

    def gwait(self, timeout=None):
        """Wait from an eventlet

        This cooperatively blocks the current eventlet by switching to
        the hub.  The hub will switch back to this eventlet when it
        gets notified.

        Usually you can just call .wait() which will dispatch to this
        method if you are in an eventlet.

        Returns True if this thread/eventlet was notified and False
        when a timeout occurred.
        """
        waiter = eventlet.getcurrent()
        hub = eventlet.hubs.get_hub()
        self._create_pipe(hub)
        self._waiters.add((waiter, hub))
        if timeout and timeout > 0:
            timeout = eventlet.Timeout(timeout)
            try:
                with timeout:
                    hub.switch()
            except eventlet.Timeout, t:
                if t is not timeout:
                    raise
                self._waiters.discard((waiter, hub))
                return False
            else:
                return True
        else:
            hub.switch()
            return True

    def twait(self, timeout=None):
        """Wait from an thread

        This blocks the current thread by using a conventional lock.

        Usually you can just call .wait() which will dispatch to this
        method if you are in an eventlet.

        Returns True if this thread/eventlet was notified and False
        when a timeout occurred.
        """
        waiter = threading.Lock()
        waiter.acquire()
        self._waiters.add((waiter, None))
        if timeout is None:
            waiter.acquire()
            return True
        else:
            # Spin around a little, just like the stdlib does
            _time = time.time
            _sleep = time.sleep
            min = __builtin__.min
            endtime = _time() + timeout
            delay = 0.0005      # 500 us -> initial delay of 1 ms
            while True:
                gotit = waiter.acquire(0)
                if gotit:
                    break
                remaining = endtime - _time()
                if remaining <= 0:
                    break
                delay = min(delay * 2, remaining, .05)
                _sleep(delay)
            if not gotit:
                self._waiters.discard((waiter, None))
                return False
            else:
                return True

    def notify(self):
        """Notify one waiter

        This will notify one waiter, regardless of whether it is a
        thread or eventlet, resulting in the waiter returning from
        it's .wait() call.

        This will never block itself so can be called from either a
        thread or eventlet itself and will wake up the hub of another
        thread if an eventlet from it is notified.
        """
        if self._waiters:
            waiter, hub = self._waiters.pop()
            if hub is None:
                # This is a waiting thread
                try:
                    waiter.release()
                except thread.error:
                    pass
            else:
                # This is a waiting greenlet
                def notif(waiter):
                    waiter.switch()
                hub.schedule_call_global(0, notif, waiter)
                if hub is not eventlet.hubs.get_hub():
                    self._kick_hub(hub)

    def notify_all(self):
        """Notify all waiters

        Similar to .notify() but will notify all waiters instead of
        just one.
        """
        for i in xrange(len(self._waiters)):
            self.notify()

    def _create_pipe(self, hub):
        """Create a pipe for a hub

        This creates a pipe (read and write fd) and registers it with
        the hub so that ._kick_hub() can use this to signal the hub.

        This keeps a cache of hubs on ``self.hubcache`` so that only
        one pipe is created per hub.  Furthermore this dict is never
        cleared implicitly to avoid creating new sockets all the time.

        This method is always called from .gwait() and therefore can
        only run once for a given hub at the same time.  Thus it is
        threadsave.
        """
        if hub in self.hubcache:
            return
        def read_callback(fd):
            # This just reads the (bogus) data just written to empty
            # the os queues.  The only purpose was to kick the hub
            # round it's loop which is now has.  The notif function
            # scheduled by .notify() will now do it's work.
            os.read(fd, 512)
        rfd, wfd = os.pipe()
        listener = hub.add(eventlet.hubs.hub.READ, rfd, read_callback)
        self.hubcache[hub] = (rfd, wfd, listener)

    def _kick_hub(self, hub):
        """Kick the hub around it's loop

        Threads need to be able to kick a hub around their loop by
        interrupting the sleep.  This is done with the help of a
        filedescriptor to which the thread writes a byte (using this
        method) which will then wake up the hub.
        """
        rfd, wfd, r_listener = self.hubcache[hub]
        current_hub = eventlet.hubs.get_hub()
        if current_hub.running:
            def write(fd):
                os.write(fd, 'A')
                current_hub.remove(w_listener)
            w_listener = current_hub.add(eventlet.hubs.hub.WRITE, wfd, write)
        else:
            os.write(wfd, 'A')

    def __repr__(self):
        return ('<gsync.Notifier object at 0x%x (%d waiters)>' %
                (id(self), len(self._waiters)))

That's all folks

So it seems that with some careful thinkering you can create all your tried and tested tools to communicate between threads and eventlets or between eventlets in different threads. This makes adopting eventlet into an existing application a whole lot more approachable. It certainly helped me!

Creating subprocesses in new contracts on Solaris 10

2011-02-09T00:30:00.001Z

Solaris 10 introduced "contracts" for processes. You can read all about it in the contract(4) manpage but simply put it's a grouping of processes under another ID, and you can "monitor" these groups, e.g. be notified when a process in a group coredumps etc. This is actually one of the tools of Solaris' init replacement smf(5), which is probably the main reason people care about contracts.

Suppose you have a daemon managed by SMF which executes subprocesses as part of it's life (e.g. because of multiprocessing). If your application is not aware of contracts all subprocesses will be in the same contract. Which is fine until the main process dies a miserable death and one expects SMF to restart it. Only it doesn't since it sees the contract as still being alive and sees no harm. Time to start your subprocesses in a different contract then.

Anyway, whatever your motivation, starting the subprocesses in new contracts is normally done by activating the process contract template, i.e. opening /system/contract/process/template and then calling ct_tmpl_activate(3) on the filedescriptor. Only problem is that this isn't exposed to python.

Fortunately the libcontract(3) functions we need for this are very simple, so ctypes can handle this very nicely. The code is pretty simple:

import ctypes
import os

fd = os.open('/system/contract/process/template', os.O_RDWR)
libct = ctypes.cdll.LoadLibrary('libcontract.so.1')
rv = libct.ct_tmpl_activate(fd)
if rv != 0:
    raise Exception('oops')

That's all there is to it. Each fork(2) call will now result in a process running in it's own contract.

It's nice when C-APIs are simple enough to be used from inside python instead of having to write wrappers.

PS: Needless to say you can do a whole lot more with contracts, just read up on the libcontract API docs.

re.search() faster then re.match()

2010-12-19T22:13:00.000Z

This is a counter-intuitive discovery, in IPython:


In [18]: expr = re.compile('foo')

In [19]: %timeit expr.search('foobar')
1000000 loops, best of 3: 453 ns per loop

In [20]: %timeit expr.match('foobar')
1000000 loops, best of 3: 638 ns per loop

So now I'm left wondering why .match() exists at all. Is it really such a common occurrence that it's worth an extra function/method?

Just to be complete, if this is actually what you want there is no performance gap:


In [25]: expr = re.compile('^foo')

In [26]: %timeit expr.search('foobar')
1000000 loops, best of 3: 617 ns per loop

In [27]: %timeit expr.match('foobar')
1000000 loops, best of 3: 612 ns per loop

Storm and SQLite in-memory databases

2010-11-02T00:10:00.000Z

When using an SQLite in-memory databases in storm the different stores created from the same database are not modifying the same SQLite database. E.g.

db = storm.locals.create_database('sqlite:')
store1 = storm.locals.Store(db)
store2 = storm.locals.Store(db)

Here store1 and store2 will not refer to the same database, despite the fact that this is what would be natural. The reason is that SQLite in-memory databases are specific to their connection object. And the connection object is part of the store object, not the database object.

The upshot is that I can't use in-memory databases inside my unittests that easily because the code under tests assumes creating stores is cheap (not caring too much about the caching). Which all kind of sucks.

PS: A whole different rant is about libraries designed for "from foo import *", e.g. storm.locals.*, fabric.api.*. At least for the later you can do "import fabric.api as fab", "import storm.locals as storm" has it's limitations...

Finding the linux thread ID from within python using ctypes

2010-09-02T13:10:00.000+01:00

So I've got a multi-threaded application and suddenly I notice there's one thread running away and using all CPU. Not good, probably a loop gone wrong. But where? One way to find this is revert history in the VCS and keep trying it out till you find the bad commit. Another way is to find out which thread is doing this, this is of course much more fun!

Using ps -p PID -f -L you'll see the thread ID which is causing the problems. To relate this to a Python thread I subclass threading.Thread, override it's .start() method to first wrap the .run() method so that you can log the thread ID before calling the original .run(). Since I was already doing all of this apart from the logging of the thread ID this was less work then it sounds. But the hard part is finding the thread ID.

Python knows of a threading.get_ident() method but this is merely a long unique integer and does not correspond to the actual thread ID of the OS. The kernel allows you to get the thread ID: getid(2). But this must be called using a system call with the constant name SYS_gettid. Because it's hard to use constants in ctypes (at least I don't know how to do this), and this is not portable anyway, I used this trivial C program to find out the constant value:

#include <stdio.h>
#include <sys/syscall.h>

int main(void)
{
    printf("%d\n", SYS_gettid);
    return 0;
}

In my case the constant to use is 186. Now all that is left is using ctypes to do the system call:

import ctypes

SYS_gettid = 186
libc = ctypes.cdll.LoadLibrary('libc.so.6')
tid = libc.syscall(SYS_gettid)

That's it! Now you have the matching thread ID!

Going back to the original problem you can now associate this thread ID with the thread name and you should be able to find the problematic thread.

Return inside with statement (updated)

2010-08-14T19:35:00.000+01:00

Somehow my brain seems to think there's a reason not to return inside a with statement, so rather then doing this:

def foo():
    with ctx_manager:
        return bar()

I always do:

def foo():
    with ctx_manager:
         result = bar()
    return result

No idea why nor where I think to have heard/read this. Searching for this brings up absolutely no rationale. So if you know why this is so, or know that the first version is perfectly fine, please enlighten me!

Update:

Seems it's only relevant if you're reading a file using the with statement. This seems to have come from the python documentation itself:

The last version is not very good either — due to implementation details, the file would not be closed when an exception is raised until the handler finishes, and perhaps not at all in non-C implementations (e.g., Jython).
def get_status(file): with open(file) as fp: return fp.readline()

Sadly it doesn't say what the implementation details are nor how to do this correctly. I can have several more or less educated guesses at why and which way to do this better. But I'd love to get a more detailed description of what happens in the implementations when doing this, because the worst-case way of interpreting that example is that using open() or file() as a context manager is completely useless. Which I would hate.

Templating engine in python stdlib?

2010-08-09T15:03:00.001+01:00

I am a great proponent of the python standard library, I love having lots of tools in there and hate having to resort to thirdparty libs. This is why I was wondering if there could be a real templating engine in the stdlib some day? I'm not a great user of templates, but sometimes string.Template is just not good enough and real templating would make things a lot more readable.

Not being a great templating user I don't know how the world of templating looks. Is there a template engine out there which would be a candidate to move to the stdlib? Or do most people think this is a stupid idea? Or maybe this has been discussed before and I didn't find the discussion?

Using Debian source format 3.0 (quilt) and svn-buildpackage

2010-07-26T22:02:00.000+01:00

Searching the svn-buildpackage manpage for the 3.0 (quilt) format I thought that it wasn't able to apply the patches in debian/patches during build time. Instead I was doing a horrible dance which looked something like "svn-buildpackage --svn-export; cd ../build-area/...; debuild". Turns out I was completely wrong.

svn-buildpackage doesn't need to know about the source format. Instead it simply invokes dpkg-buildpackage which will automatically notice that the patches are not applied and apply them before building. That simple!

Thanks to Niels Thykier to point this out to me on IRC.

Europython, threading and virtualenv

2010-07-23T14:32:00.000+01:00

I use threads
correctly

I do not use virtualenv
and dont't want to

Just needed to get that out of my system after europython, now mock me.

PS: I should probably have done this as a lightening talk but that occurred to me too late.

py.test test generators and cached setup

2010-06-13T16:44:00.005+01:00

Recently I've been enjoying py.test's test function arguments, it takes a little getting used too but soon you find that it's quite likely a better way then the xUnit-style of setup/teardown. One slightly more advanced usage was using cached setup together with test generators however. While not difficult that took me some figuring out, so let me document it here.

Since I haven't been a fan of generative tests before I'll explain why I think I can make use of them now. I was writing a wrapper around pysnmp to handle SNMP-GET requests transparently between the different versions. For this I wrote a number of test functions which do some GET requests and check the results, the basic outline of such a test is:

def test_some_get(wrapper_v1):
    oids = ...
    result = wrapper_v1.get(oids)
    assert ...

Here wrapper_v1 is a funcarg which returns an instance of my wrapper class configured for SNMPv1. The extra catch here is that this funcarg uses a function which tries to find an available SNMP agent, trying if one is running on the local host (for the developer) or if a well-know test host is reachable (for lazy developers on our dev network and for buildbots), skipping the test otherwise. But to avoid the relatively long timeouts involved for each individual test this function needs to be cached. Here's the outline of this funcarg:

def pytest_funcarg__wrapper_v1(request):
    cfg = request.cached_setup(setup=check_snmp_v1_avail, scope='session')
    if not cfg:
        py.test.skip('No SNMPv1 agent available')
    return SnmpWrapper(cfg)

Once having all the tests using this wrapper_v1 funcarg I obviously want exactly the same tests for SNMPv2 since that's the whole point of the wrapper. For this I'd need a wrapper_v2 funcarg which is configured for SNMPv2, but that would mean duplicating all the tests! Enter test generators.

The trick to combine test generators with cached setup is not to use the funcargs argument to metafunc.addcall() but rather use the param argument in combination with a normal funcarg. The normal funcarg can then use request.cached_setup() and use the request.param to decide how to configure the wrapper object returned. This is what that looks like:

def pytest_generate_tests(metafunc):
    if 'snmpwrapper' in metafunc.funcargnames:
        metafunc.addcall(id='SNMPv1', param='v1')
        metafunc.addcall(id='SNMPv2', param='v2')

def pytest_funcarg__snmpwrapper(request):
    cfg = request.cached_setup(setup=lambda: check_snmp_v1_avail(request.param),
                               scope='session', extrakey=request.param)
    if not cfg:
        py.test.skip('No SNMP%s agent available' % request.param)
    return SnmpWrapper(cfg)

Don't forget the extrakey argument to cached_setup. The caching uses the name of the requested object, "snmpwrapper" in this case, and the extrakey value to decide when to re-use the caching. If you forget extrakey both calls will return the same cfg.

And that's all that's needed! Test now simply ask for the snmpwrapper funcarg and will get run twice, once configured for SNMPv1 and once for SNMPv2. Running the tests will now look like this:

flub@signy:...$ py.test -v snmp_test.py
============================= test session starts ==============================
python: platform linux2 -- Python 2.6.5 -- pytest-1.1.1 -- /usr/bin/python
test object 1: /home/flub/.../snmp_test.py

snmp_test.py:57: test_get_one.test_get_one[SNMPv1] PASS
snmp_test.py:57: test_get_one.test_get_one[SNMPv2] PASS
snmp_test.py:63: test_get_two.test_get_two[SNMPv1] PASS
snmp_test.py:63: test_get_two.test_get_two[SNMPv2] PASS
snmp_test.py:71: test_get_two_bad.test_get_two_bad[SNMPv1] PASS
snmp_test.py:71: test_get_two_bad.test_get_two_bad[SNMPv2] PASS
snmp_test.py:79: test_get_many.test_get_many[SNMPv1] PASS
snmp_test.py:79: test_get_many.test_get_many[SNMPv2] PASS

=========================== 8 passed in 1.31 seconds ===========================

This wasn't very complicated, but having an example of using the param argument to metafunc.addcall() would have made figuring this out a little easier. So I hope this helps someone else, or at least me at some time in the future.

Update: Originally I forgot the extrakey argument to cached_setup() and thus the funcarg was returning the same in both cases. Somehow I assumed the caching was done on function identity of the setup function. Oops.

Selectable queue

2010-05-29T14:34:00.001+01:00

Sometimes you'd want to use something like select.select() on Queues. If you do some searching for this it turns out that this question has been answered for multiprocessing Queues where you can simply use the real select on the underlying socket (IIRC), but for a good old fashioned Queue you're stuck.

Now it's easy to argue that this isn't that high a need, when I wanted this a while ago it turned out to be surprisingly simple to re-structure the design a little so that I no longer desired a selectable queue. But it's still something that hung around the back of my mind for a while, so I've kept thinking about it. My conclusion for now (which I haven't bothered implementing) is that simply cloning the normal queues but replacing the not_empty and not_full Conditions by Events gives you selectable queues. It changes the semantics slightly, but that doesn't seem harmful. This obviously isn't enough, so the second change to the queue is that you should be allowed to pass in the events to use. And now that you can share the not_full event between two queues you can simply wait on this event and you have your select.

Next time I want this I might actually implement this idea rather then re-design so that I don't want selectable queues anymore.

weakref and circular references: should I really care?

2010-05-12T19:09:00.001+01:00

While Python has a garbage collector pretty much whenever circular references are touched upon it is advised to use weak references or otherwise break the cycle. But should we really care? I'd like not to, it seem like something the platfrom (python vm) should just provide for me. Are all those mentions really just for the cases where you can't (or don't want to, e.g. embedded) use the normal garbage collector?

Update: By some strange brain failure I seemed to have written "imports" rather then "references" in the title originally. They are obviously a bad thing.

python-prctl

2010-05-08T14:19:00.000+01:00

There is sometimes a need to set the process name from within python, this would allow you to use something like "pkill myapp" rather then the process name of your application just being yet another "python". Bugs (which I'm now failing to find in Python's tracker) have been filed about this and many wannabe implementations made, all of them seemed to have many problems however. Aiming for UNIX cross-compatibility they all messed up and end up trying to make the abstraction everywhere, and ususally the code was less then beautiful

But I've just discovered the python-prctl module by Dennis Kaarsemaker which does something far more sensible: rather then trying to be cross-platform it just wraps the Linux prctl system call (as well as libcap). The code looks well written and the API, while a little bit overloaded on the get-set names, seems nice. It even includes what is probably the most sensible implementation of clobbering argv that I've ever seen (but don't use that, no normal person should ever clobber argv!).

If someone writes a nice module like this to cater for the MacOSX guys, the only other system I know of has a system call to set the process name, then I may never have to worry about getting someting like this into PSI at some distant point in the future. (And speaking about PSI, yes the windows port is still slowly under way. I"m just busy with lots of other things at the same time.)

Storm and sqlite locking

2010-05-08T02:44:00.000+01:00

The Storm ORM struggles with sqlite3's transaction behaviour as they explain in their source code. Looking at the implementation of .raw_execute() the side effect of their solution to this is that they start an explicit transaction on every statement that gets executed. Including SELECT.

This, in turn, sucks big time. If you look at sqlite's locking behviour you will find that it should be possible to read from the database using concurrent connections (i.e. from multiple processes and/or threads at the same time). However since storm explicitly starts that transaction for a select it means the connection doing the read now holds a SHARED lock until you end the transaction (by doing a commit or rollback). But since it's holding this shared lock, for no good reason, it means no other connection can acquire the EXCLUSIVE lock at the same time.

The upshot of this seems to be that you need to call .commit() even after just reading the database, thus ensuring you let go of the shared lock. Can't say I like that.

Python optimisation has surprising side effects

2010-04-27T20:43:00.001+01:00

Here's something that surprised me:

a = None
def f():
    b = a
    return b
def g():
    b = a
    a = 'foo'
    return b

While f() is perfectly fine, g() raises an UnboundLocalError. This is because Python optimises access to local variables using the LOAD_FAST/STORE_FAST opcode, you can easily see why this is looking at the code objects of those functions:

>>> f.__code__ .co_names
()
>>> f.__code__ .co_varnames 
('a', 'b')
>>> g.__code__ .co_names
('a',)
>>> g.__code__ .co_varnames 
('b',)

I actually found out this difference thanks to finally watching the Optimizations And Micro-Optimizations In CPython talk by Larry Hastings from PyCon 2010. I never realised that you could create a situation where the nonlocal scope would not be looked in.

Judging performance of python code

2010-04-03T00:40:00.000+01:00

Recently I"ve been messing around with python bytecode, as a result I now know approximately how the python virtual virtual machine works at a bytecode level. I've found this quite interesting but other then satisfying my own curiosity had no benefit from this knowledge. Until today I was wondering which code would be more efficient:

a += 1
if b is not None:
    a += 2

if b is None:
    a += 1
else:
    a += 3

From a style point of view I'd prefer to write the first: I find it slightly more readable since it's got fewer indented blocks and it uses one line less on my screen. But my gut feeling tells me the second is faster, particularly if b is not None (this because in the first sample we'd do 2 add operations instead of one if b is not None). But now I can verify my gut feeling! All I need to do is compile both fragments and use the disassembler to investigate:

>>> co1 = compile(sample1, '', 'exec')
>>> dis.dis(co1)
  2           0 LOAD_NAME                0 (a) 
              3 LOAD_CONST               0 (1) 
              6 INPLACE_ADD          
              7 STORE_NAME               0 (a) 

  3          10 LOAD_NAME                1 (b) 
             13 LOAD_CONST               2 (None) 
             16 COMPARE_OP               9 (is not) 
             19 POP_JUMP_IF_FALSE       35 

  4          22 LOAD_NAME                0 (a) 
             25 LOAD_CONST               1 (2) 
             28 INPLACE_ADD          
             29 STORE_NAME               0 (a) 
             32 JUMP_FORWARD             0 (to 35) 
        >>   35 LOAD_CONST               2 (None) 
             38 RETURN_VALUE
>>> co2 = compile(sample2, '', 'exec')
>>> dis.dis(co2)
  2           0 LOAD_NAME                0 (b) 
              3 LOAD_CONST               2 (None) 
              6 COMPARE_OP               8 (is) 
              9 POP_JUMP_IF_FALSE       25 

  3          12 LOAD_NAME                2 (a) 
             15 LOAD_CONST               0 (1) 
             18 INPLACE_ADD          
             19 STORE_NAME               2 (a) 
             22 JUMP_FORWARD            10 (to 35) 

  5     >>   25 LOAD_NAME                2 (a) 
             28 LOAD_CONST               1 (3) 
             31 INPLACE_ADD          
             32 STORE_NAME               2 (a) 
        >>   35 LOAD_CONST               2 (None) 
             38 RETURN_VALUE

My analysis of this is pretty simple: count the number of instructions for when b is None and for when b is not None.

	b is None	b is not None
sample1	10	15
sample2	11	10

So ultimately the best performance depends on whether b will be None or not. However the difference in the best case is only one instruction, but the difference in the worst case is a whole mighty 4 instructions! This would seem to confirm my gut feeling: the ugly code is better. It also makes me wonder if this is not the sort of optimisation a compiler should be doing: create the bytecode for sample2 regardless of the source code (I'm not a compiler guy and do realise it might not be as simple as that since python is a dynamic language which allows you to change stuff, including executed code, at runtime, yada yada).

There's one more catch tough: I seriously doubt each python instruction can be executed in the same time! So let's use timeit to actually verify this. I'm omitting the trivial code, but this is the result:

$ python3 test.py 
sample1, b is None: 0.128307104111
sample2, b is None: 0.128338098526
sample1, b is not None: 0.244062900543
sample2, b is not None: 0.12109208107

To be honest, the result is exactly as speculated: the first sample is slower when b is not None, all others are pretty much the same. The one odd thing is the pretty much doubling of time for the bad case, this suggest the python virtual machine is spending most of the time doing the INPLACE_ADD instruction while all others are probably very quick.

Anyway, in conclusion I guess you can speculate about performance and get an idea by knowing what bytecode will be generated. But at the end of the day you'll get a simple and better answer by simply using timeit. So knowing something about python bytecode still hasn't gained me any benefit. It was still an interesting exercise tough.

Hacking mock: Mock.assert_api(...)

2010-04-01T21:45:00.000+01:00

Mock is a great module to use in testing, I use it pretty much all the time. But one thing I have nerver felt great about is the syntax of it's call_args (and call_args_list): it is a 2-tuple of the positional arguments and the keyword arguments, e.g. (('arg1', 'arg2'), {'kw1': None, 'kw2': None}). This does show you exactly how the mock object was called, but the problem I have is that it's more restrictive then the signature in python:

def func(foo, bar, baz=None):
    pass

func(0, 1, 2)
func(0, 1, baz=2)
func(0, bar=1, baz=2)
func(foo=0, bar=1, baz=2)

In this example all the calls to func() are exactly the same from python's point of view. But they will all be different in mock's call_args attribute. To me this means my test will be too tightly coupled to the exact implementation of the code under test. This made me wonder how it could be better.

Firstly I started out writing a function which would take the call_args tuples and know the function signature. This obviously isn't very nice as you need a new function for each signature. But this lead me on to adding the .assert_api() method to the mock object itself. I've been using this method for a while and am still not disappointed in it, so tought I should write about it. Here's how to use it:

mobj = mock.Mock(api='foo,bar,baz')
do_stuff()
mobj.assert_api(foo=0, bar=1, baz=2)

It seems to me this is a fairly good compromise to an extra method on the mock object (and attribute, not shown) and nice concise way of asserting if a function was called correctly.

There are a number of side effects, mainly due to my implementation. The major one is that it consumes .call_args_list! This means each time you call .assert_api() the first item disappears from .call_args_list. Again, I like this as it allows me easily to check multiple calls by just doing multiple .assert_api() calls.

Another side effect, of less importance, is a new attribute "api" on the mock object, I can imagine people disliking that one. But it's never been in my way and means you can just assign to it instead of using the keyword argument when creating the mock. I find this handy in combination with patch decorators where the syntax to fine-tune the mock is rather heavy to my liking.

The last strange thing is .assert_api_lazy(), which is just a horribly bad name. It ignores any arguments which are present in the call on the mock object, but where not passed in as part of the api='...' parameter. It's effectively saying you only want to check a few of the arguments and don't care about the others.

Finally here is the code, for simplicity here implemented (and unchecked) as a subclass of Mock:

class MyMock(Mock):
    def __init__(api=None, **kwargs):
        Mock.__init__(self, **kwargs)
        self.api = api

    def _assert_api_base(self, **kwargs):
        """Return call_kwargs dict for assert_api and assert_api_lazy

        WARNING, this consumes self.call-args_list
        """
        if self.call_args_list:
            call_args, call_kwargs = self.call_args_list.pop(0)
        else:
            raise AssertionError('No call_args left over')
        call_args = list(call_args)
        if self.api is None:
            raise AssertionError('self.api is not initialised')
        for p in self.api.split(','):
            if call_args:
                call_kwargs[p] = call_args.pop(0)
        return call_kwargs

    def assert_api(self, **kwargs):
        """WARNING, this consumes self.call_args_list"""
        call_kwargs = self._assert_api_base(**kwargs)
        assert kwargs == call_kwargs, \
            'Expected: %s\nCalled with: %s' % (kwargs, call_kwargs)

    def assert_api_lazy(self, **kwargs):
        """WARNING, this consumes self.call_args_list"""
        call_kwargs = self._assert_api_base(**kwargs)
        for k in call_kwargs.copy().iterkeys():
            if k not in kwargs:
                del call_kwargs[k]
        assert kwargs == call_kwargs, \
            'Expected: %s\nCalled with (truncated): %s' % (kwargs, call_kwargs)

I'd be interested in feedback! It's definitely not perfect, but I do like the syntax of m=Mock(api='...'); ...; m.assert_api(...). One problem I can think of is that it won't deal gracefully with default arguments on the api yet, e..g.:

def func(foo, bar=42):
    pass

func(foo, 42)
func(foo)

These two calls are identical, but .assert_api() won't see them as the same. This hasn't bothered me yet, which is why I haven't looked into it. But I guess it should be considered for a general-purpose implementation of this idea.

Using lib2to3 in setup.py

2010-03-26T07:40:00.000Z

It seems that many people think you need distribute if you want to run 2to3 automatically in your setup.py. But personally I don't like setuptools (aka distribute) and hence don't like forcing this on users. No worries since plain old distutils supports this as well, but it simply appears less well known.

All you need to do is use the build_py_2to3 command supplied by distutils.command.build_py in python3 instead of the normal build_py command. This is how you can do this:

try:
    from distutils.command.build_py import build_py_2to3
except ImportError:
    pass

COMMANDS = {}
if sys.version_info[0] == 3:
    COMMANDS['build_py'] = build_py_2to3
setup(..., cmdclass=COMMANDS, ...)

That's it! Ok, you have to do slighty more then just add a keyword argument to the setup() call. But just the 2to3 feature is not worth using distribute for.

Discovering basestring

2010-02-10T20:26:00.001Z

This may seem simple and trivial, but today I discovered the basestring type in Python. It is essentially a base type for str and unicode, which comes in handy when writing isinstance(foo, basestring) in test code for example.

Strangely, despide PEP 237 mentioning the equivalent for int and long as "integer" I can't actually find what has become of that, so still writing isinstance(foo, (int, long)) for now.

Note that this is Python 2.x only, in 3.x it's all unicode so there is no longer a need for a common base type. int and long are unified as well in Python 3.x obviously.

Registering callbacks: to quack like a duck or tell which bit will quack the duck way?

2010-02-05T16:20:00.000Z

When you have someplace where you register a set of callbacks you have a few options:

Use an interface-style API. I.e. you have a Mixin style class defining the methods expected to be present, probably just raising NotImplementedError. You then just pass in the entire object to the .register() function.
Just try if it quacks like a duck. IMHO this is just the pythonic equivalent of the above, simply without the explicit mixin class.
Explicitly pass in the callback functions. This would be something like: .register(func1, ...). Secret benefit here is using the result of functools.partial() and things like it.

What I don't like about the first two options is that you force the method name of the callback onto the user. The third option allows you to achieve the same but at the cost of a more complicated register function. But I tend to prefer it because it's more explicit and it avoids "hiding" things in the object (via the mixin) thus keeping the interfacace between caller and callee cleaner.

Are there other design considerations when making this choice? Maybe there's already an essay somewhere exploring these ideas in more detail then I'm doing here?

Pointer arithmetic in C

2010-01-25T10:53:00.000Z

Pointer arithmetic in C is great. The only downside is that it's only defined when pointing to an array (or an item just after the array). That is one giant downside.

Allocate a chunk of memory and you can't use pointer arithmetic. So if you are building a linked list this means you will have to allocate each item in the list separately, even if you know the total length the list will ever have.

Even K&R must have realised this was an annoying limitation. They point this out when they show how to write malloc (K&R2 8.7 Example -- A Storage Allocator)

There is still one assumption, however, that pointers to different blocks returned by sbrk can be meaningfully compared. This is not guaranteed by the standard, which permits pointer comparisons only within an array. Thus this version of malloc is portable only among machines for which general pointer comparison is meaningful.

If only the standard would have relaxed this restriction slightly (e.g. to continuously allocated regions) life would be so much better

The master plan

2010-01-18T21:19:00.000Z

Catherine Devlin describes it very nicely:

I finally understand Al-Qaeda's master plan, and it's freaking brilliant. [...] I'm just surprised that we're choosing to participate in the plan. I thought we were on opposite sides?

Terminology

2010-01-10T18:02:00.003Z

Why was it ever considered desirable to call a directory containing a __init__.py file a "package" rather then just "module". They are after all simply "modules containing other modules". It's not like that solved some sort of problem was it? But, as sad as that sometimes might be, we can't change the past.

Anyway, people are continiously confused when talking about python and all the module distributions made available on ~~the cheeseshop~~PyPI by various projects. And now yet another discussion rages over at distutils-sig where they want to re-work this confusion by using unambigous terms. The only possible outcome I can think of is more confusion.

What is so hard about accepting the current status quo? The thing that would help most is documenting something like the following in distutils documentation:

The word "package" is often used to refer to a module distribution, usually it is perfectly clear form the context whether it is talking about a module distribution or a module containing other modules.

Installing Debian on a Sun Fire X2200 M2

2010-01-09T16:00:00.001Z

Most important thing first: the eLOM console lives on ttyS1 (9600n8). If that gets you going you do not need to read any more.

This works pretty easily, but there's isn't much information about how this works if you don't like using a keyboard and a screen (having these things appears to be the norm for x86 hardware). But the X2200 comes with this "Embedded Lights Out Manager" which, although not as good as the LOM on a T1000, does the job nicely. It allows you to connect via the serial console, IPMI and ssh. So initially I used the serial console to setup the IP configuration of the eLOM and then I could ssh to it (and more importantly sit in a quiter room far away from the rack).

Once inside the eLOM it allows you to connect to the "serial port" of the machine (with the rather obsusucre "start /SP/AgentInfo/console" command, disconnect by hitting Esc + Shift-9). This allows you to see the BIOS settings etc. Now is the time to insert the Debian CD (or netboot or whatever), if inserting the CD you'll just get a blank screen since it's displaying pretty graphics, hitting escape gives you the boot prompt as explained in the Debian installation manual. Now the serial console you're seeing will appear as the second serial port to the OS (this is configurable in the BIOS I think but I didn't play with those settings). So the correct way to start the installation is:


expert fb=false console=ttyS1,9600n8

(or "install" instead of "expert" if you prefer that)

This will start the normal install process you're used too (and which is described in other guides) with the output appearing on your terminal emulator. One tip tough is to enable the installation via ssh when it allows you to select the installer components to load. The serial console is 9600 baud and painting the dialog windows over that is just painfully slow, this is a lot nicer over ssh.

The last thing to note is that debian-installer is smart enough to enable a getty on the serial console you've used for the installation (if fact it doesn't enable any gettys on the normal ttys). But it doesn't append the console=ttyS1,9600n8 to the grub menu, so you'll want to do this yourself if you like seeing the kernel output on the console.

Decorators specific to a class

2009-12-30T13:22:00.003Z

My gut tells me it's horribly wrong but I am failing to formulate a decent argument, let me show an example what I mean (somewhat contrived):

def deco(f):
    def wrapper(self, *args, **kw):
        with self.lock:
            f(self, *args, **kw)

class Foo:
    def __init__(self):
        self.lock = threading.Lock()

    @deco
    def method(self):
        pass

Here the decorator knows something about the arguments of the function it will end up calling, the first argument is "self" and it is an object with a "lock" attribute which is a context manager. Somehow I feel like that's more knowledge about the wrapped object then a decorator should have. It just is an indirection of logic my bain doesn't cope with.

There are obviously places where I could construct something like this. But I do never naturally think of doing it that way, I always end up with some other way which I find more elegant and I think the resulting logic is easier to follow. It's just that whenever I encounter code like this my brain starts hurting and I'm not sure I have a decent argument to tell people writing code like this off (you can hardly regard "it's a level of logic indirection that makes my brain hurt" as an argument).

Calling COM methods in C

2009-12-26T12:05:00.000Z

So Windows has this strange thing called COM. It allows you to share objects between unrelated processes and languages, a bit like CORBA in that sense, only very different. Anyway, if you'd like to get information out of this other Windows thing called WMI (which provides a lot of information you can't get hold of using native APIs, at least not if you don't digg into unpublished internals) then you've got to use COM.

But Microsoft seems to have decided some long time ago that C++ is such an amazing language (because if it's got ++ in the name it must be better then say just C, you know) that it solves all of the world's problems. So obviously that is the language you favour when designing APIs, meaning you end up with very convoluted APIs when using good old plain C. And obviously there is no need to document the way you do things in C because no one would use it. Anyway, my conclusion is that COM is crazy and I can only assume that .NET must somehow make this a bit easier (just like using WMI from Python is easy with Mark Hammond's pythoncom and Tim Golden's wmi module) which is probably the reason that C# is popular among developers who choose Windows as their platform.

Back to the point however: how do you call a method on a COM object using C? Microsoft will always show instance->Method() in their examples, but in C it's not that much different:

instance->lpVtbl->Method(instance)

There is also supposed to be a way to use some macro's using the class name of the object joined up with the method name. For this you need to define the COBJMACROS before your #includes and then you can call:

ClassName_Method(instance)

Only I can't get that way to work. No clue why.

For completeness here is the full example of the WMI local computer example in the MSDN library, ported to C. For extra fun the error handling is done by setting Python exceptions, but that shouldn't confuse things. It also prints the result string in both unicode and ascii, from the unicode string you could have easily made a PyUnicodeObject. There's probably many ugly things in there as I'm really not a win32 developer. It's also example code and not what I'd take into production (e.g. I think the looping over the results is broken but I'm sticking close the the MSDN example).

static int
example(void)
{
    HRESULT hr;
    IWbemLocator *pLoc;
    IWbemServices *pSvc;
    IEnumWbemClassObject *pEnumerator;
    BSTR bstr, bstr2;

    hr = CoInitializeEx(0, COINIT_APARTMENTTHREADED);
    if (hr == RPC_E_CHANGED_MODE)
        hr = CoInitializeEx(0, COINIT_MULTITHREADED);
    if (hr != S_OK && hr != S_FALSE) {
        PyErr_Format(PyExc_WindowsError,
                     "Failed to initialise COM (HRESULT=0x%x)", hr);
        return -1;
    }
    hr = CoInitializeSecurity(NULL, -1, NULL, NULL,
                              RPC_C_AUTHN_LEVEL_DEFAULT,
                              RPC_C_IMP_LEVEL_IMPERSONATE,
                              NULL, EOAC_NONE, NULL);
    if (FAILED(hr)) {
        PyErr_Format(PyExc_WindowsError,
                     "Failed to initialise COM security (HRESULT=0x%x)", hr);
        CoUninitialize();
        return -1;
    }
    hr = CoCreateInstance(&CLSID_WbemLocator, 0, CLSCTX_INPROC_SERVER,
                          &IID_IWbemLocator, (LPVOID *) &pLoc);
 
    if (FAILED(hr)) {
        PyErr_Format(PyExc_WindowsError,
                     "Failed to get IWBemLocator (HRESULT=0x%x)", hr);
        CoUninitialize();
        return -1;
    }
    bstr = SysAllocString(L"ROOT\\CIMV2");
    if (bstr == NULL) {
        PyErr_SetString(PyExc_WindowsError, "BSTR allocation failed");
        pLoc->lpVtbl->Release(pLoc);
        CoUninitialize();
    }
    hr = pLoc->lpVtbl->ConnectServer(pLoc, bstr, NULL, NULL,
                                     NULL, 0, NULL, NULL, &pSvc);
    SysFreeString(bstr);
    if (FAILED(hr)) {
        PyErr_Format(PyExc_WindowsError,
                     "Localhost connection for WMI failed (HRESULT=0x%x)", hr);
        pLoc->lpVtbl->Release(pLoc);
        CoUninitialize();
        return -1;
    }
    hr = CoSetProxyBlanket((IUnknown*)pSvc, RPC_C_AUTHN_WINNT, RPC_C_AUTHZ_NONE,
                           NULL, RPC_C_AUTHN_LEVEL_CALL,
                           RPC_C_IMP_LEVEL_IMPERSONATE, NULL, EOAC_NONE);
    if (FAILED(hr)) {
        PyErr_Format(PyExc_WindowsError,
                     "Failed to set ProxyBlanket (HRESULT=0x%x)", hr);
        pSvc->lpVtbl->Release(pSvc);
        pLoc->lpVtbl->Release(pLoc);
        CoUninitialize();
        return -1;
    }
    bstr = SysAllocString(L"WQL");
    if (bstr == NULL) {
        PyErr_SetString(PyExc_WindowsError, "BSTR allocation failed");
        pSvc->lpVtbl->Release(pSvc);
        pLoc->lpVtbl->Release(pLoc);
        CoUninitialize();
        return -1;
    }
    bstr2 = SysAllocString(L"SELECT * FROM Win32_OperatingSystem");
    if (bstr2 == NULL) {
        PyErr_SetString(PyExc_WindowsError, "BSTR allocation failed");
        SysFreeString(bstr);
        pSvc->lpVtbl->Release(pSvc);
        pLoc->lpVtbl->Release(pLoc);
        CoUninitialize();
        return -1;
    }
    hr = pSvc->lpVtbl->ExecQuery(pSvc, bstr, bstr2,
                                 WBEM_FLAG_FORWARD_ONLY |
                                 WBEM_FLAG_RETURN_IMMEDIATELY, 
                                 NULL, &pEnumerator);
    SysFreeString(bstr);
    SysFreeString(bstr2);
    if (FAILED(hr)) {
        PyErr_Format(PyExc_WindowsError, "WMI query failed (HRESULT=0x%x)", hr);
        pSvc->lpVtbl->Release(pSvc);
        pLoc->lpVtbl->Release(pLoc);
        CoUninitialize();
        return -1;
    }
    {
        IWbemClassObject *pclsObj;
        ULONG uReturn;
        VARIANT vtProp;       
            
        while (pEnumerator) {
            hr = pEnumerator->lpVtbl->Next(pEnumerator, WBEM_INFINITE,
                                           1, &pclsObj, &uReturn);
            if(uReturn == 0)
                break;
            hr = pclsObj->lpVtbl->Get(pclsObj, L"Name", 0, &vtProp, 0, 0);
            wprintf(L"XXX OS Name: %s\n", vtProp.bstrVal);
            {
                /* XXX Need error checking in here */
                /* Allocating the UTF-16 string size since that will be at
                 * least double the ASCII size, which is fine. */
                char *prop;
                int r;

                prop = psi_malloc(SysStringByteLen(vtProp.bstrVal));
                /* if (prop == NULL) */
                r = WideCharToMultiByte(20127, 0, vtProp.bstrVal, -1, prop,
                                        SysStringByteLen(vtProp.bstrVal),
                                        NULL, NULL);
                /* if (!r) */
                printf("XXX OS Name: %s\n", prop);
            }
            VariantClear(&vtProp);
            pclsObj->lpVtbl->Release(pclsObj);
        }
    }
    pEnumerator->lpVtbl->Release(pEnumerator);
    pSvc->lpVtbl->Release(pSvc);
    pLoc->lpVtbl->Release(pLoc);
    CoUninitialize();
    return 0;
}

Setting descriptors on modules

2009-12-23T16:59:00.000Z

This counts for one of the more crazy things I'd like to do with Python: insert an instance of a descriptor object into the class dict of a module.

While you can get hold of the module type class by using the __class__ attribute of any module or use types.ModuleType you obviously can't do this since the __dict__ of the module class is actually a DictProxy and hence immutable. Which I think is rather sad this time round.

My use case is to be able to set an attribute on a module that would lazily evaluate a sort of semi-singleton. Suppose you have the instance of your application and to make it available to other modules you place the instance in a module attribute. You want it to be available since it has references to useful global things like the configuration instance in use etc (and you hate having singletons for all these useful things). To now get this application instance from another module, without getting it passed in with some sort of dependency-injection (which can result in intangible spaghetti all too easily), you can now get hold of it like this:

import app_package.app_module
app_package.app_module.app_instance # the instance

But what you very likely can't do is this:

from app_package.app_module import app_instance

The reason is that import statements are usually at the top of modules and thus you will very likely execute this import statement before the app_instance existed and you will get None (I will anyway, since I initialised that attribute with None before the app gets instanced). And obviously even once the application is instanced and the attribute gets set, I'm still stuck with the reference to None instead of the application instance.

So my reason to want to set a descriptor instance in the module class is so that the descriptor could lazily evaluate the application instance when accessing it. Getting and using the instance would reduce to something like:

from app_package.app_module import app_instance
app_instance # the instance!

Which I think is much cleaner. But for now I'll have to stick with:

from app_package.app_module import get_instance
get_instance() # the instance

Or perhaps Python has another trick up it's sleeve I haven't found yet?

Skipping slow test by default in py.test

2009-12-19T20:33:00.000Z

If you got a test suite that runs some very slow tests it might be troublesome to run those all the time. There is of course the "-k" option to py.test which allows you to selectively enable only a few tests. But what I really wanted was a way to have it skip slow tests by default but still allow me to enable slow tests with a command line option.

But this is not impossible, py.test provides a couple of things that make it possible to do this surprisingly easy:

You can add command line options
You can mark tests as with whatever name you like
You can skip tests

So how do you pull this together? I decided that i want to mark slow tests using the "slow" keyword (@py.test.mark.slow) and skip those tests by default. And the option to enable those tests would be called "--slow". The following conftest.py is staggeringly simple:

import py.test

def pytest_addoption(parser):
    parser.addoption('--slow', action='store_true', default=False,
                      help='Also run slow tests')

def pytest_runtest_setup(item):
    """Skip tests if they are marked as slow and --slow is not given"""
    if getattr(item.obj, 'slow', None) and not item.config.getvalue('slow'):
        py.test.skip('slow tests not requested')

Finding this solution was a little harder however, but essentially it involved nothing more then looking at the pytest_skipping and pytest_mark plugins to figure out what APIs the various objects provide. But it must be said that extending py.test is staggeringly simple, I tend to expect a much higher learning curve when I decide that I want to write a plugin for some tool I use.

setuptools vs distribute

2009-12-18T01:51:00.000Z

Reinout van Rees:

In case you heard of both setuptools and distribute: distribute fully replaces setuptools. Just use distribute. Setuptools is “maintained” (for various historically dubious values of “maintain”) by one person (whom all should applaud for creating the darn thing in the first place, btw!). Distribute is maintained by a lot of people, so bugs actually get fixed. And “bugs” meaning “it doesn’t break with subversion 1.6 because patches that fix it don’t get applied for half a year”.

Pretty much one of the best descriptions of the differences between setuptools and distribute that I've seen. Not that I use either, but whatever.

Dictionary entry

2009-12-07T18:29:00.001Z

innovative company noun 1 a company which can't afford patent lawyers 2 one of the lucky handful companies who are innovative and have money for patent lawyers HELP Not to be confused with most companies lobying in patent litigation.

PS: It's also possible to have companies that can't afford patent lawyers but still don't innovate.

How to drive on a motorway

2009-12-06T21:47:00.000Z

If you are driving on a motorway where I am driving too, here are some rules you should follow. I think you should even follow them when I'm not around.

Be sensible, this one overrules all the others
Keep left unless overtaking
KEEP LEFT UNLESS OVERTAKING
KEEP LEFT UNLESS OVERTAKING
Always indicate when changing lanes
Pick a speed and stick to it, avoid speeding up or slowing down whenever possible
Be courteous, even when people make mistakes

It's not that hard, try it and you'll realise traffic would be a lot more fluent if everyone stuck to these rules.

PS: Don't forget to substitute left with right depending on your country.

How to make a crosslinked RS-232 cable

2009-12-03T19:33:00.000Z

Because it's easier then buying one

When connecting two computes together (e.g. your laptop to a server or router) to get access to a console you need to use a crosslinked RS-232 cable, usually with two female DE-9 connectors these days. This cable is more commonly known as a null-modem cable for historical reasons.

Of course you can easily buy a cable under the name of "null modem cable", but you're almost guaranteed to get a sloppy made one. In fact I haven't been able to find a decently made one. The problem is that according to the standard the DTR is supposed to be connected to the DSR and CD, but most cables as sold won't connect the carrier detect line. No idea why, perhaps making the bridge is something other connectors don't need and therefore expensive to automate for production?

So to make one yourself you can cut up an existing cable and solder new DE-9 connectors onto it yourself, keeping to the proper schema. It's really not that hard, all you need is a soldering iron and two D-sub 9 pin connectors and corresponding hoods. I'm guessing you could even use cheap CAT-5e networking cable instead of cutting up an expensive pre-made null-modem cable since Sun uses network cables for their serial cables (whith an adaptor).

Now most devices actually seem to ignore the carrier detect line anyway. But in case you are lucky enough to be using IBM's AIX or IVM on one of their System p5 servers you will discover that you are able to log into the service processor with a "normal" (i.e. non-standard) serial cable just fine. You will also be able to perform an installation from that searial cable just fine. But when you then need to access the console of that just installed system you're stuck and just see nothing. Until you use a proper cable that is. (Just for the record, you can most likely telnet or ssh into the system by now, just try it).

And if my love for AIX is not yet clear from the above paragraph let me be a bit more explicit: if you are in a position to choose get something else then AIX. Sun's hardware as well as Solaris are lovely (they ship with a serial cable for instance - as does cisco). Obviously GNU/Linux is a great choice for an OS too. This doesn't mean that I approve of cable vendors not making RS-232 crosslink cables propperly of course.

On using .getstate()

2009-12-01T18:51:00.000Z

Do not, I repeat, do not define .__getstate__() idly.

Thank you very much

Tuple unpacking goodness

2009-11-29T14:35:00.000Z

Todays pleasant surprise:

>>> a = {'a': (0, 1), 'b': (2, 3)}
>>> for k, (v1, v2) in a.iteritems():
...     print k, v1, v2
... 
a 0 1
b 2 3
>>>

Nice!

The new faces of Europe

2009-11-28T13:57:00.000Z

I can't help but agree.

Finding memory leaks in extension modules

2009-11-25T00:07:00.000Z

After reading Ned Batchelder's post on his experience hunting a memory leak (which turned out to be a reference counting error) it occurred to me that even tough I have a script to check memory usage I should also really be checking reference counts with sys.gettotalrefcount(). And indeed, after adding this to my script I found one reference count leak. I still have faith in my script as it was before really since the reference leak in question was not making me loose memory - subtle bugs eh?

But how do you check an extension module for memory leaks? This seems pretty undocumented so here my approach:

First you really need a debug build of python, this helps a lot since you get to use sys.gettotalrefcount() and get more predictable memory behaviour. The most complete way to build this is something like this (the MAXFREELIST stuff adapted from this):
s="_MAXFREELIST=0" ./configure --with-pydebug --without-pymalloc --prefix=/opt/pydebug \ CPPFLAGS="-DPyDict$s -DPyTuple$s -DPyUnicode$s -DPySet$s -DPyCFunction$s -DPyList$s -DPyFrame$s -DPyMethod$s" make make install

Now run the test suite using valgrind, this is troublesome but a very useful thing to do. The valgrind memory checker will help you identify problems pretty quickly. It can be confused about Python however, but you only care about your extension module so you need to filter most of this. Luckily the python distribution ships with a valgrind suppression file in Misc/valgrind-python.supp that you can use, it's not perfect but helps. This is how I invoke valgrind:

$ /opt/pydebug/bin/python setup.py build
$ valgrind --tool=memcheck \
    --suppression=~/python-trunk/Misc/valgrind-python.supp \
    --leak-check=full /opt/pydebug/bin/python -E -tt setup.py test
==8599== Memcheck, a memory error detector
==8599== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==8599== Using Valgrind-3.5.0-Debian and LibVEX; rerun with -h for copyright info
==8599== Command: /opt/pydebug/bin/python -E -tt setup.py test
==8599== 
==8599== Conditional jump or move depends on uninitialised value(s)
==8599==    at 0x400A66E: _dl_relocate_object (do-rel.h:65)
==8599==    by 0x4012492: dl_open_worker (dl-open.c:402)
==8599==    by 0x400E155: _dl_catch_error (dl-error.c:178)
==8599==    by 0x4011D0D: _dl_open (dl-open.c:616)
==8599==    by 0x405AC0E: dlopen_doit (dlopen.c:67)
==8599==    by 0x400E155: _dl_catch_error (dl-error.c:178)
==8599==    by 0x405B0DB: _dlerror_run (dlerror.c:164)
==8599==    by 0x405AB40: dlopen@@GLIBC_2.1 (dlopen.c:88)
==8599==    by 0x8132727: _PyImport_GetDynLoadFunc (dynload_shlib.c:130)
==8599==    by 0x81199D9: _PyImport_LoadDynamicModule (importdl.c:42)
==8599==    by 0x81161FE: load_module (import.c:1828)
==8599==    by 0x8117FAF: import_submodule (import.c:2589)
...
running test
...
FAILED (failures=4, errors=2)
==8599== 
==8599== HEAP SUMMARY:
==8599==     in use at exit: 1,228,588 bytes in 13,293 blocks
==8599==   total heap usage: 280,726 allocs, 267,433 frees, 70,473,201 bytes allocated
==8599== 
==8599== LEAK SUMMARY:
==8599==    definitely lost: 0 bytes in 0 blocks
==8599==    indirectly lost: 0 bytes in 0 blocks
==8599==      possibly lost: 1,201,420 bytes in 13,014 blocks
==8599==    still reachable: 27,168 bytes in 279 blocks
==8599==         suppressed: 0 bytes in 0 blocks
==8599== Rerun with --leak-check=full to see details of leaked memory
==8599== 
==8599== For counts of detected and suppressed errors, rerun with: -v
==8599== Use --track-origins=yes to see where uninitialised values come from
==8599== ERROR SUMMARY: 75 errors from 5 contexts (suppressed: 19 from 6)

Note that the output is very verbose, usually I actually start with --leak-check=summary. Firstly notice that valgrind gives a lot of warnings already before your extension module gets loaded, that's python's problems and not yours so skip over that. The stuff output after (and during) the output of the test suite is what interests you. Most importantly look at the definitely lost line if that's not zero the you have a leak. The possibly lost is just python's problem (which sadly might hide problems you created too). When you do have lost blocks valgrind will give you a stack trace to pinpoint it, but you'll have to swim trough lots of "possibly lost" stack traces of python to find it. Best is probably to grep for your source files in the output.

Next you should create function you want to execute in a loop, this should be exercising the code you want to tests for leaks. If you're really thourough possibly the entire test suite wrapped up in a function call would be good.

Wrap it all up in a script that checks the memory usage and reference counts on each loop and compares the start and end values. Getting memory usage might be tricky from python (or you can use PSI of course) so depending on your situation you might prefer to do this with an script from your operating system.

For PSI this is the script I currently use. I clearly have it easy since I can be sure PSI will be available :-). The reason I don't automate this script further (you could turn it into a unittest) is that I prefer to manually look at the output. Both memory and reference counting are funny and will most likely grow a little bit anyway. By looking at the output I can easily spot if it keeps growing or stabilises, there is only a problem if it keeps growing with every iteration (don't be afraid to run with many many iterations from time to time). When automating this you probably end up allowing some margin and might miss small leaks.

Hopefully some of this was useful for someone.

New Python System Information release!

2009-11-21T15:59:00.000Z

I've just released a new version of PSI! PSI is a cross-platform Python package providing real-time access to processes and other miscellaneous system information such as architecture, boottime and filesystems. Among the highlights of this release are:

Improved handling of time: We now have our own object to represent time. This may seem silly at first but it actually makes it easier to use all the normal ways of representing time easily as well as provide the highest possible accuracy.
Improved handling of process names and arguments: There is an entire wiki page dedicated to this, but basically this simplifies presenting sensible process names to users massively since some attributes will always have meaningful values.
Restructured exceptions: Whenever accessing attributes you will get a subclass of AttributeError like it should be so now you can happily use getattr().
New experimental methods on Process objects: You can now send signals to processes using the .kill() method and find their children by using the .children() method.
New experimental mount module: You can get detailed information about all mounted filesystems using this new module. It provides mount information as well as usage.

Another notable improvement is the ability to read the arguments of a 64-bit process while running inside a 32-bit python process on Solaris. It's small and almost no-one will notice it but make it so much more consistent!

Release early, release often: fail

Now for the bad news: all this means the API has changed in a backwards incompatible way.

It was already pretty obvious shortly after the last release that this would happen and was the reason I was hoping to release a new version soon. But that didn't happen. Although the last version had a "beta" version number on it it's trove classifier still claimed to be "alpha" and in end we don't promise API stability till we hit 1.0. But it's still not nice. Once we hit 0.3 we will actually try not to introduce changes to the API if possible. We intend to help this by using FutureWarning for APIs we're not sure about yet. In the mean time let's see how the Process API holds out during this release, hopefully it will prove to be good and require no more changes.

Credits

As before, thanks to Chris Miles and Erick Tryzelaar for helping out.

Synchronous classes in Python

2009-11-16T13:39:00.000Z

What I'd like to build is an object that when doing anything with it would first acquire a lock and release it when finished. It's a pattern I use fairly-regularly and I am getting bored of always manually defining a lock next to the other object and manually acquiring and releasing it. It's also error prone.

Problem is, I can't find how to do it! The __getattribute__ method is bypassed by implicit special methods (like len() invoking .__len__()). That sucks. And from the description of this by-passing there seems to be no way to get round it. For this one time where I thought I found a use for meta-classes they still don't do the trick...

Python modules and the GPL: I still don't get it

2009-11-03T22:59:00.001Z

I've never understood if you can use GPL python modules (or packages) in GPL-incompatibly licensed code. Today I re-read the GPL looking for this and am tempted to think yes, you can. The GPL says:

1. You may copy and distribute verbatim copies of the Program's source code as you receive it [...]

This is simple enough, what if you need to change it? This gets more interesting:

2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, [...] provided that you also meet all of these conditions:

[...]

b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.

[...]

These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.

As I uderstand this it means you can happily ship a GPL python module next to a GPL-incompatible module, even when the second uses API calls from the first. As long you are always offering to give the source of the GPL module, including modifications if you made any, you are fine. The two modules are individually identifiable sections and can be reasonably considered independent and separate works, or so I reckon at least.

But wait, the non-GPL module uses the API of the GPL module, is it still independent and separable then? In my humble opinion the GPL says it must be reasonably considered independtent and separate, so yes. It's feasable to take away the GPL module and replace it by another one that offers the same API, hence I would argue they are separable (feasable and easy are not synonyms!).

According to my reasoning it should also be legal to use GPL C libraries in non-GPL programs as long as you use them as shared libraries and not as a static library. But why does the Lesser General Public License (LGPL) exist then? Qoting the the LGPL preamble (emphasis mine):

When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library.

Now a shared C library is linked by the dynamic linker before the application gets started. If the library is not present the application will fail horribly, I guess this is why they consider it legally speaking a combined work. When using a python module the python interpreter will happily start and depending on how the application is written can fail gracefully or even provide partial functionality.

The GPL FAQ argues that it depends on the technical means used to run the final application whether the modules should both be GPL or not. The arguments seem to suggest that if code runs in shared address space as GPL-licensed code, then the whole must comply to the GPL. Given that translation of a work is considered a derrivative we can assume python translates the python code (interprets it) and then executes the results in the same address space, therefore all must comply to the GPL (including python itself).

I'm still hard pushed to call the non-GPL module a derrivative work of the GPL module however. And in a lenghty article (apparenlty written by real lawyers, definitely worth a read!) the authors argue that legally it depends on a lot more then the technicality to determine if a work becomes a derrivative work or not (and that's where it all revolves around: if its a derrivative and you distribute it the GPL applies). This does indeed seem a lot more logical: imagine you accept the interpretation in the GPL FAQ, all you need to do to distribute the two modules is use pyro or some other form of inter-process communication (excluding shared memory according to the GPL FAQ, this again I find hard to accept) and use the APIs of the modules over this IPC layer.

The above article describes how some courts have judged this derrivation question and gives a couple of pointers itself. Essentially it common sense, but that's hard to qantify as even the GPL FAQ fails. I should really summarise the factors in a paragraph here, but I was formally trained to be an engineer and not a lawyer so am not used to summarising lawyer articles (and frankly I'm too lazy to perform the exercise right now). Beside I'd probably skip a lot of subtle points so you're best off reading the article yourself.

I do have a conclusion however: if you have a python module which is not GPL-compatible but uses the API of another python module covered by the GPL chances are you are fine if: (i) you are not taking away market share of the GPL module. And (ii) you are not derriving or extending the creative work or copyrightable content of the module. But there's no distinct line, common sense is your friend (and enemy).

Delny 0.4.1

2009-10-24T20:59:00.001+01:00

A little while ago I released a new version of Delny (my python wrapper around Qhull for Delaunay triangulations), the main purpose to use numpy instead of numeric. Impressively enough people actually seemed to care and I got a few bug reports and hints for improvements.

So I just released 0.4.1 with some of these updated:

I forgot to change some python code to numpy, so was still importing numeric
Use numpy.get_include() to find the numpy header files

At the same time I fixed the 2D square issue by using the Qz option and removing the extra point from all output. Finally fixing a unittest that has been failing for ages.

To compensate however I added a new test that fails to triangulate a 3x3 2D square (well, it works but doesn't create simplical facets), thanks to Stephen McQuay's bug report. AFAIK there's not bug for this in Delny's code, Qhull just seems to return the wrong result (but it does work on the command line).

Now if someone would donate me windows binaries that would save me answering those questions too...

No permission to see Ubuntu bugs?

2009-10-22T23:47:00.000+01:00

So I'm looking at the release notes for Ubuntu 9.10 and am interested in the hibernation issue. Naturally I follow the link to the bug report.

First I need to log in to launchpad. That's weird, since when do I need to log in simply to view bugs? But after logging in I see this:

Not allowed here
Sorry, you don't have permission to access this page.
You are logged in as Floris Bruynooghe.

Since when is Ubuntu hiding bugs? Sure security bugs might need to be hidden for a while but you could be more descriptive about that. And surely you shouldn't be linking to such bugs?

Just weird.

Cross platform shell scripts

2009-09-24T21:18:00.000+01:00

At work we have an rather elaborate collection of shell scripts that builds our softweare and all it's dependencies on about 7 UNIX variants as well as on Windows. Shell seemed like a good choice when we started to write those scripts: it is available on all hosts (using Cygwin on Windows) and with some care you can do most task in a portable way, ssh and scp are at your fingertips to do things on remote hosts without any extra hassle etc.

The thing I didn't realise is just how extrordinary expensive forking is on Windows. And when having lots of shell that's what you're doing all the time (cut, tr, grep, awk, ...), on UNIX you generally don't even notice it but on Windows it just grinds to a halt.

This basically means I'm slowly moving to replacing the scripts in Python. Each time I need to modify something I consider the cost of re-writing it in Python and of keeping it in shell. Some critical parts are already replaced by Python code and the speedup is impressive.

So next time you need cross-platform scripts, think about how much work it will need to do on Windows. Forking is expensive. Sadly.

Battery life

2009-09-23T08:07:00.002+01:00

Normally I'm quite happy with the 3h of battery life of my laptop, it covers all my disconnected time on trains etc. But it just doesn't cut it on a 10h flight, especially painful if my brain has loads of ideas to try out and things to do.

New Delny release

2009-09-20T17:50:00.004+01:00

A few days ago I got another of those two-a-year inquiries about Delny, the python wrapper around Qhull for Delaunay triangulations. But increadably indirectly it finally got me to do a new release, it's only been a few years since the last one! Nothing has changed really, the only difference it that it finally uses numpy to compile instead of the deprecated numeric

I've also taken the opportunity to move the revision control to mercurial and host it on bitbucket. And uploaded the tarball to PyPI, not sure why I didn't do that before.

Hope this keeps being of use to some people.

Resuming an scp file transfer

2009-08-20T12:03:00.003+01:00

Sometimes you have to transfer data over dodgy links. And when you're transferring a large file this can hurt when it fails. So as explained elsewhere rsync can safe the day:

$ scp host:remote_file local_file
# now interrupt and resume with
$ rsync --partial --progress --rsh=ssh host:remote_file local_file

You need rsync on the remote server for this however. But usually that's not too much of a hurdle.

Should bare except statements be allowed in the Python stdlib?

2009-08-13T11:46:00.003+01:00

Firstly to clarify the terminology, this is bare except statement:

try:
    ...
except:
    ...

And this is a non-bare except statement, but bear in mind the type of the exception that is caught can be anything:

try:
    ...
except Exception:
    ...

The point is that both fragments are a catch-all exception handler, only the second is slightly more better/restrictive since it won't catch a SystemExit exception for example (which you rarely want to catch). This is obviously discussed before and even made it to a (rejected) PEP.

So I'm tempted to say that the stdlib should not use bare except statements. If you need to catch more then Exception they can always catch BaseException. However grepping the stdlib reveals 384 cases of bare except statements, to be fair many of these are in test cases but still.

The one that hurt me today was in socketserver.BaseServer.handle_request, now I have to re-write the .handle_error() function to call sys.exc_info() and check that it's a subclass of Exception before handling the error normally. That's not nice.

Pain

2009-08-04T00:27:00.003+01:00

I've spent the last 2 days trying to get a stack trace from a crashing python extension module in windows. And I still haven't figured it out. That's sooo very motivating.

Give me GNU/Linux any day.

cifs why don't you follow unix conventions?

2009-08-03T22:07:00.004+01:00

Often it's nice to just have conventions and gentleman's agreements. But then sometimes someone doesn't and the harm is done, it's too hard to reverse.

NFS has introduced the de-facto standard of using "<host>:<path>" for the defice of a remote mount. Yes this break because you can have ":" in a filename, but in practice it works.

<rant>

But somehow cifs (formerly smbfs), decided that adhering to the windows syntax was more important then adhering to the unix standard, never mind that it's a filesystem implementation for unix-like systems. So the remote mountpoint for a cifs filesystem is "//<host>/<path>". Yes, this breaks too, you could start any path with "//". But it's just plain annoying when trying to figure out if a filesystem is remote or not.

</rant>

Letters on keyboard stop working

2009-07-30T15:49:00.003+01:00

Dear Lazy Web

I have some Very weird behaviour where the lowercase letters "C" and "V" stop generating the Correct key press event after a short while (which is why I am typing them as upper Case here...). But only when using Compiz as window manager, when switching away from X to the Console they work again but in X they don't. When looking at the keypress in xeV it shows up like this:

FocusOut event, serial 34, synthetic NO, window 0x5000001,
    mode NotifyGrab, detail NotifyAncestor

FocusIn event, serial 34, synthetic NO, window 0x5000001,
    mode NotifyUngrab, detail NotifyAncestor

KeymapNotify event, serial 31, synthetic NO, window 0x0,
    keys:  82  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
           0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0

For Comparison with a normal keypress:

KeyPress event, serial 31, synthetic NO, window 0x5000001,
    root 0xa9, subw 0x0, time 18685100, (888,414), root:(898,513),
    state 0x10, keycode 53 (keysym 0x78, x), same_screen YES,
    XLookupString gives 1 bytes: (78) "x"
    XmbLookupString gives 1 bytes: (78) "x"
    XFilterEvent returns: False

KeyRelease event, serial 34, synthetic NO, window 0x5000001,
    root 0xa9, subw 0x0, time 18685275, (888,414), root:(898,513),
    state 0x10, keycode 53 (keysym 0x78, x), same_screen YES,
    XLookupString gives 1 bytes: (78) "x"
    XFilterEvent returns: False

I am totally at a loss what the problem Could be. And searching for this is really hard and so far entire unsuccessful. The most suspicious thing I Can find in Xorg.log is:

(WW) Logitech USB Multimedia Keyboard: unable to handle keycode 267

But I have no idea if this is related.

Any hints?

PyCon UK 2009

2009-07-28T12:56:00.003+01:00

So there's a PyCon UK 2009 Unconference. I wish I could go but have already planned to be on Pabbay and Mingulay in the Outer Hebrides, Schotland. So instead of hearing all about python I'll be clinging on to their amazing cliffs. Guess I'm not that bad off anyway...

How to bring a running python program under debugger control

2009-07-25T14:06:00.002+01:00

Of course pdb has already got functions to start a debugger in the middle of your program, most notably pdb.set_trace(). This however requires you to know where you want to start debugging, it also means you can't leave it in for production code.

But I've always been envious of what I can do with GDB: just interrupt a running program and start to poke around with a debugger. This can be handy in some situations, e.g. you're stuck in a loop and want to investigate. And today it suddenly occurred to me: just register a signal handler that sets the trace function! Here the proof of concept code:

import os
import signal
import sys
import time


def handle_pdb(sig, frame):
    import pdb
    pdb.Pdb().set_trace(frame)


def loop():
    while True:
        x = 'foo'
        time.sleep(0.2)


if __name__ == '__main__':
    signal.signal(signal.SIGUSR1, handle_pdb)
    print(os.getpid())
    loop()

Now I can send SIGUSR1 to the running application and get a debugger. Lovely!

I imagine you could spice this up by using Winpdb to allow remote debugging in case your application is no longer attached to a terminal. And the other problem the above code has is that it can't seem to resume the program after pdb got invoked, after exiting pdb you just get a traceback and are done (but since this is only bdb raising the bdb.BdbQuit exception I guess this could be solved in a few ways). The last immediate issue is running this on Windows, I don't know much about Windows but I know they don't have signals so I'm not sure how you could do this there.

devenv.com via cygwin ssh (visual studio 2003)

2009-07-13T16:48:00.002+01:00

Integrating an automatic build on a windows host with the rest of your one-command cross-platform build system can be quite a pain. Luckily there is cygwin which makes things like ssh, bash, make etc all work on windows. It's great.

The trick to building using visual studio from there is to use the devenv.com tool, which is a version of the full blown visual studio (devenv.exe) that does not pop up windows on your screen (ahem, should not- see below) but instead shows all output nicely on your terminal (which you tee to the logfiles of your build system of course). Life still looks good.

So you set up your build system to do remote, unattended logins to the windows build slave using ssh public keys. This is trivial as you do this on all your build slaves. But all of a sudden devenv.com just hangs. That's weird. Do some searching on the mighty Internet and it turns out there's a bug somewhere in Mircosofts authentication handling. There is some token that does something weird and somehow when you use public key authentication visual studio 2003 (and 2005) think they're the system service (since that is what the ssh service runs as) and don't run properly (various errors possible). But everyone reports errors instead of just hanging, how is this possible? So you go to the ssh service and tick the "Allow service to interact with desktop" box, restart ssh and try again. Success! Now you get a window from devenv.com that visual studio crashed on the screen. Never mind that devenv.com was supposed to be command-line only.

But still not closer to the solution. It turns out that Microsoft actually fixed this error for visual studio 2005, but you're stuck with 2003 for some reason.

Time for ugly solutions.

Since this Windows box is a build slave anyway, it's not supposed to be used for anything else. So the only account of use on it is the one used by the build system. What if we run the ssh service in the name of this account? This requires some fiddling with the permissions of the ssh files (/etc/ssh_* and /var/log/sshd.log) in cygwin, but once the ssh service is happy and wants to start it all works! No more errors from devenv.com

Ugly, but it finally works.

Importing modules in C extension modules

2009-07-05T14:11:00.002+01:00

It seems that if you need another module in a function of your extension module, the way modules in the standard library seem to solve this is like this:

static PyObject *
func(void)
{
    PyObject *foo;

    foo = PyImport_ImportModuleNoBlock("foo");
    if (foo == NULL)
        return NULL;
    /* do stuff with foo */
    Py_DECREF(foo);
    return something;
}

This means that you have to import the module each time you enter the function (yes, it's looked up in the modules dict by PyImport_ImportModuleNoBlock() but that function is only avaliable since 2.6, before you have to use PyImport_ImportModule()).

Personally I like storing the module in a static variable so that it only needs to be imported the first time:

static PyObject *
func(void)
{
    static PyObject *foo = NULL;

    if (foo == NULL) {
        foo = PyImport_ImportModuleNoBlock("foo");
        if (foo == NULL)
            return NULL;
    }
    /* do stuff with foo */
    return something;
}

Note here that the Py_DECREF() is gone. This function will effectively leak a reference to the module object. But is this really bad? How often do module objects get deleted in production code? My guess is that they normally stay loaded until the application exits.

Singletons in a Python extension module

2009-06-23T08:46:00.003+01:00

If you want a singleton in a C extension module for CPython you basically have to do the same as when doing this in plain Python: the .__new__() method needs to return the same object each time it is called. Implementing this has a few catches though.

static PyObject *
MyType_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    static MyObject *self = NULL;

    if (self == NULL)
        self = (MyObject *)type->tp_alloc(type, 0);
    Py_XINCREF(self);
    return (PyObject *)self;
}

Then assign this function to the tp_new slot in the type. There's two things of interest in this function:

self is declared as static. Normally this would not be the case, but declaring it as static makes it stay alive after the function has returned and it will still be pointing to the first instance of our object, so the next time a new object is asked for this is simply returned.
Before returning the pointer to the static self, the reference count is increased. This may seem odd but is the right thing to do, otherwise the reference count will not go up for subsequent calls to the new function since the reference count is increased by PyType_GenericAlloc() (via the call from the pointer in the tp_alloc slot). So if you don't do this you end up with negative reference counts, which doesn't make python very happy. This does mean you never end up deallocating the object since the lowest reference count you have is 1, but you wanted a singleton right?. If you really wanted to get the reference count to drop to 0 then you can always put the Py_INCREF() in an else clause.

Voting and identification in the UK

2009-06-07T12:27:00.004+01:00

Last Thursday I had the pleasure of being able to vote for the local council as well as for the European Parliament. Since I'm a Belgian citizen living in the United Kingdom this involved convincing the officials at the voting office to re-read their instructions (at first they only allowed me to vote for the local council but not for Europe, I think they must have realised how silly that sounded) but otherwise was quite easy. Too easy.

When I went to the polling station I forgot my poll card, but my colleagues at work said that would be fine (not that going home to pick it up was very far) so I tried it anyway. No problem, the only thing they asked is where I lived, then found my name on their list and let me vote (after above-mentioned debacle). There was absolutely no verification that I was who I claimed to be. Seriously, it is rather trivial to vote for someone else, I'm sure you can visit 2 or 3 polling stations and vote in someone else their name. Just talk to friends and work a little bit together and you can all cast half a dozen votes if you're a little careful. And they have no way of recovering from this, other then having to let everyone using that voting station re-cast their vote.

After some talking to people they seemed to think that there is no single means of identifying someone in an official way in the UK. This means that buying alcohol is better controlled then voting, since there you either have a form of ID (if you look under 21) or you don't get to buy it. But for voting? No ID required, because there is none.

I'm sure the ID card scheme as currently proposed in the UK is not any good, but there does appear to be a problem that might have to be fixed somehow. By now I'm pretty sure that if you give me 2 years I can create a fictitious John Smith person, he'll have a passport, driving license, voting rights, bank account and be native British. Seriously, it's easy. You just need some time.

Raising exceptions in threads

2009-06-06T13:05:00.003+01:00

It's not simply raising an exception in a thread that I want. I want to raise an exception in a thread from another thread. It's like sending signals to threads, only signals in pyhon can only be delivered to the main thread (for portability). But Python has a other asynchronous signaling system: exceptions.

I'd like to be able to do something like:

for t in threading.enumerate():
    thread.raise(MyAppDoesntWantYouAnymoreError)

Is this possible? Are there other ways to do this sort of thing?

Alternatively I might be happy with a fix for issue1856, but I do think it might be nice to be able to signal threads in an asynchronous way in any case.

New home for PSI

2009-05-30T15:04:00.003+01:00

PSI, the Python System Information package that pulls interesting information from the kernel and makes it available in a nice Python API, has a new home at bitbucket. This means that the source conde now lives inside a mercurial repository instead of subversion.

This actually happened about a week ago, but better announce it late then never...

Python System Information 0.3b1^W0.3b1.1

2009-05-21T10:04:00.000+01:00

Short summary: PSI is alive and we've just released the first beta of a much improved upcoming 0.3 release!

Back in 2007 Chris Miles announced PSI - Python System Information, a Python extension module to provide access to some system information not normally available to Python. Most notably it allows you to look at all processes on the system and get details like memory usage, cpu usage, users and many more things the kernel knows about a process. And all this in a pythonic way!

At the time the implementation was not perfect (when will an implementation ever be?), it had many memory leaks and reference counting errors, and sadly no one seemed to have the time and motivation to work on it for a long time. But since the beginning of this year I finally found some time to fix these issues and soon some more people joined in. Chris has been amazing in allowing access to almost anything I asked for and since then there has been steady development improving the code base, tests and API. Right now PSI does not leak any memory, and provides basic system indentification and detailed process information on Linux (2.4 & 2.6 kernels), Darwin 10.3 and up, SunOS (5.8-5.11) and AIX (5.3) and it does this for any version of Python greater then 2.3, including 3.0. Not too shabby!

There are many things we would like to do in the future too. more platform support for one. Ideally PSI should run on all the major platforms supported by Python itself (and all minor ones too). And more information too, getting information about processes is one very useful and common thing but there is so much more the kernel can tell us: CPU information and statistics, network interfaces, etc. It's a massive and never ending task, but hopefully we can do the common things on all major platforms.

Update

Version 0.3b1.1 has been released now. Seems the last version did only ship the sources for Linux and not all platforms. This bugfix release also adds the MANIFEST file so that distutils can build binary distributions from the tarball.

Sorry!

Compiling applications in setup.py

2009-05-11T20:16:00.000+01:00

If you need to compile a random application in setup.py this is not that hard:

cc = distutils.ccompiler.new_compiler()
distutils.sysconfig.customize_compiler(cc)
cc.link_executable(['test.c'], 'a.out')

There is no need to create the object files explicitly with cc.compile() first if you have no need for them, the command line invoked by .link_executable() will do the compilation step in one go for you.

This part of distutils is actually documented, so check it out, you can pass in many optional arguments and modify the compiler object to customize things.

What's new in Python 2.6: logging

2009-05-01T22:45:00.000+01:00

The What's New in Python 2.6 document is very good and contains loads of information, congrats for making it. But it misses out a nice addition to the logging module: LoggerAdapter Objects. It allows you to create a logger object with extra contextual information really easy:

log = logging.LoggerAdapter(logging.getLogger("subsystem"),
                            {"argv1": sys.argv[1]})
logging.basicConfig(format="%(argv1)s %(message)s")

Neat

Secondly it decided to define __all__ which means not all object are exported anymore. Unfortunately this __all__ is incomplete in 2.6.2, even getLogger is not listed in it! The good news is that this is one of the fastest bug report turnaround times I have seen, so expect this to be fixed in the next stable release.

Sun Fire T1000 not closet friendly

2009-04-16T17:52:00.002+01:00

Obviously the Sun Fire T1000 is a noisy machine, that's why it's sitting in the closet in the first place. But it has more shortcomings that make it a closet-unfriendly server:

It shuts down at 50 degrees Celsius. That means wrapping it in a duvet to damp the noise is not a great idea if you want it to do any work.
The power button sticks out. That makes it way to easy to press it if you're getting something else from the closet.

Time, floating points and python modules handling time

2009-03-28T21:33:00.004Z

Some memory in the back of my mind tells I've once read a rant about storing time that argued you should never store time in floating point variables. My memory seems to think the rant convinced me and it does indeed seem better to store time in integers so that time doesn't go and change little bits due to rounding etc. and indeed the C library seems to stick to that, mostly (difftime() returns a double for example).

When looking at Python the stdlib datetime module seems to do this too. However ofter people scoff at the datetime module and recommend the use of mxDateTime instead, it is a lot better supposedly. But looking at how it stores time it seems to use double values interally.

So I am wondering, if mxDateTime gets away with storing time as floating points is there really a disadvantage to it? Is there a point to avoiding floating point numbers while handling time?

Reading a 64-bit core from a 32-bit proc on solaris?

2009-03-28T16:15:00.003Z

I'm trying to read some pointers from a 64-bit address space file (from /proc/pid/as) while running in a 32-bit process - all this on Solaris. To do this I'm using the transitional large file compilation environment, i.e. where you get both open()/pread() and open64()/pread64() since I still want to be able to read a 32-bit core. However when doing the pread64() call on the address space I keep getting EOVERFLOW, no matter what values I put in - even trying to read the first byte of the core. And reading the first byte of a 64-bit core using simply pread(), which as far as I can tell should work, fails too. EOVERFLOW whatever I do.

Is this simply Solaris stopping me from reading a core from a 64-bit proc? But why wouldn't it stop me at the time I do the open64() call in that case? Any clues would be appreciated.

Replacing the screen of your digital camera

2009-03-23T23:19:00.002Z

The screen of my Canon Digital Ixus 40 compact camera broke on my last trip (sport climbing in Geyikbayiri near Antalya, Turkey - if you climb: it's an amzing place!) which made me rather sad. But with the help of Ebay I found a new screen and have just managed to replace it.

It only took about 1h40 of careful fiddling and figuring out how the camera was stuck together while being nervous of breaking it even more. But the end result was successful!

How to send emails

2009-03-23T20:32:00.002Z

Dear World

Subject lines in emails are there for a reason.
Please use them.

Love
Floris

Virtual memory size (VSZ) on AIX

2009-02-28T16:13:00.003Z

If you have ever looked at the ps(1) output on an AIX box and wondered why the virtual memory size (VSZ) is so small, often even smaller then the resident set size (RSS), then here's the answer: it's not the virtual memory size. It is actually the size of the data section of the virtual memory of the process.

If you really do want to know the VSZ of a process you'll have to use svmon -P 123, there you will find it. But do multiply this by the pagesize (pagesize(1) is a user tool on AIX). If only somehow this would be documented.

Oh, and if someone knows where the source of svmon lives I'd love to know how they actually find that, using the struct procent64 structure from procinfo.h I can not figure out a way of finding it. And the VSZ information in /proc/123/psinfo is completely whacky, no idea what that is supposed to be.

For completeness sake: this is on AIX 5.3.

Closed source frustration

2009-02-26T14:56:00.004Z

Developing against a closed source libraries is jut painful and wastes your time. Function definitions are missing from the header files, so you get to declare them yourself from the manpages. Then you get random error messages even though you do everything just like the manpage tells you. Some dark corners of the internet then suggest one of the parameters is actually the 32-bit variant of the type instead of the generic one. Still no luck.

Just let me look inside the code to figure out why it's not working, thank you very much.

This post was brought to you by the joys of the getprocs64(3) call on AIX 5.3. Any hints on it's usage would be appreciated.

Update: It finally works!

ssh magic

2009-02-19T20:42:00.002Z

Dear Lazy Web

If I write the following in my ~/.ssh/config:

Host bartunnel
  HostKeyAlias bar
  HostName bar
  LocalForward 8822 foo:22

Host barjump
  HostKeyAlias bar
  HostName localhost
  Port 8822

Then I can connect to host bar via host foo (circumnavigating a firewall that stops me from going to bardirectly) just like am connecting to it directly. E.g. in two separate shells (in this order):

$ ssh bartunnel # this sets up the tunnel
# different shell (or use -n on the last one)
$ ssh barjump # now I'm connected normally

Now is there something I could write in my ssh configuration file that I could just do this in one step? I want to simply do:

$ ssh barjump

and the tunnel should be set up for me in the background. Likewise if I close the connection the tunnel should go. Is this possible?

Planning upgrades

2009-02-15T12:32:00.002Z

So Debian has finally released again. Congratulations. Time to start planning upgrades of servers then.

Compiling 32-bit Python on amd64

2009-02-13T23:18:00.000Z

If you ever feel the need to compile a 32-bit version of Python on a amd64 bit machine, this is how you do it on a Debian/Ubuntu system.

Firstly you need the correct compiler stuff, this means you need gcc-multilib and 32-bit development libraries of at least libc. On Debian/Ubuntu installing the gcc-multilib package will pull in most if not all of the required dependencies.

Next is invoking the configure script of Python. Sadly Python is one of those autoconf using projects who advertise the use of environment variables like CFLAGS in the --help output of ./configure but don't actually respect them, this is all too common for autoconf-but-no-automake using projects. So the correct way to start building is using the OPT environment variable instead of CFLAGS:

OPT=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/pym32
make

You may want to finetune the OPT value, since this is where normally -g -O3 etc appears in so you've just got rid of those. I'm not quite convinced of this design but anyway.

Now you can watch the compilation, depending on your machine you may have time to make a cup of tea or so. Near the end you'll see something like this:

Failed to build these modules:
_hashlib           _sqlite3           _ssl
_tkinter           bz2                gdbm

This is pretty much exactly the same as when normally compiling python: go find the packages that provide the development libraries needed for these modules. Only this time you need to look for the 32-bit variety of these. On Debian/Ubuntu look out for packages named like lib32foo-dev. After installing all applicable ones I could find this is what I got it down too in the end (only using system packages):

Failed to build these modules:
_hashlib           _sqlite3           _ssl
_tkinter           gdbm

Just in case you aren't quite happy with your achievement so far you could now try compiling an extension module against your 32-bit python:

LDFLAGS=-m32 /opt/pym32/bin/python setup.py build

Now wasn't this useful? Silly binary-only libraries...

sed for binary files: bbe

2009-02-13T10:59:00.003Z

While GNU sed takes care not to munge NULL characters this is not the POSIX standard, therefore it's no surprise some implementations don't manage to edit binary files properly (not to mention that many implementations will struggle with long lines). Hence the search for a binary sed:

bbe is just that. It allows you to do operations on blocks --you can define the block size in a variety of flexible ways-- as well as on bytes. So the most simple substitution command from sed just looks exactly the same. It's gorgeous.

It is a C program that you'll need to compile, but has no build dependencies so you can use it pretty much anywhere.

Specifying the RUNPATH in sofiles

2009-02-09T18:49:00.002Z

When you create a shared object that depends on another shared object there are multiple ways to tell it where to look for the shared object. Two of these are by encoding (at linking time) a DT_RPATH and DT_RUNPATH entry in the .dynamic section. A third one is to use the LD_LIBRARY_PATH environment variable (at runtime). The order of precedence of these is:

RPATH
LD_LIBRARY_PATH
RUNPATH

The question is how to encode these into your shared object. This is normally done with the --rpath or -R options to the linker. But as the name suggest this will create an RPATH. When using GNU ld(1) you add an --enable-new-dtags option which does add the (newer) RUNPATH, and when both RPATH and RUNPATH are present the runtime linker will ignore the RPATH. On Solaris however the linker does add a RUNPATH by default as soon as you use -R, thus you will always end up with LD_LIBRARY_PATH overriding the value you gave with -R. Good to keep that in mind.

Resistance to change

2009-01-31T22:05:00.002Z

Why can C developers be happy with using config.h to get constants about where to look for configuration or data files for example, yet Python developers seem to refuse to think of any way they might want to support finding these files in a portable way?

FreeBSD on Virtualbox

2009-01-31T14:50:00.004Z

Virtualbox is my desktop virtualisation technology of choice most of the time and I wanted to have a play with FreeBSD. Seems they don't get along tough and you won't get a network connection.

Solution 1: Change the network adaptor in Virtualbox to PCnet-PCI II (instead of PCnet-PCI III). I've tried this and it works.
Solution 2: I haven't tried this.

While we're on the subject, when running the Ubuntu 8.10 server jeos edition on Virtualbox don't forget to enable PAE in Virtualbox or you get to do a kernel dance using the resque mode of the installer.

datetime.datetime.totimestamp()

2009-01-27T11:25:00.004Z

There exists a datetime.datetime.fromtimestamp() and many a time I've wondered why there is no .totimestamp() equivalent. The answer is that not all datetime objects can be represented by a timestamp. But if you know this restriction and want it anyway, this is the magic:

time.mktime(datetime_object.timetuple())

Would be so cool if the docs actually mentioned this.

PS: Both functions have a utc variant too

Update:

datetime_object.strftime('%s')

This solution does not give you sub-second resolution, but otherwise is rather elegant. Funny thing is that the %s specifier is not documented by the stdlib, but seems to exist on the underlying implementations at least on UNIX.

And for completeness, these issues are being discussed in the bug tracker. See issue2736 and issue1673409.

Resident Set Size (RSS) from /proc/pid/stat

2009-01-22T17:52:00.002Z

Most UNIX-like systems have information about processes stored in /proc/pid/ files, so does Linux.

If you would want to get the Resident Set Size (RSS) of a process on Linux you could find this in a number of files:

/proc/pid/stat: targetted for scanf(3)
/proc/pid/statm: targetted for scanf(3) but just memory information (more detailed)
/proc/pid/status: targetted at humans

Oddly enough if you check these three files for the RSS memory you will get different results! It seem both stat and statm have the wrong RSS information. It is a mystery to me why.

Porting a C extension module to py3k

2009-01-15T22:22:00.002Z

This was not that hard! First look at what everyone else has to say about this, nice aggregated on the python wiki. It covers a lot.

Now for what I discovered on top of this:

When you're defining a new type self->ob_type doesn't exist anymore. This is a problem as you need it in the deallocator for example. The solution is to use Py_TYPE(self). So the deallocator becomes: Py_TYPE(self)->tp_free((PyObject*)self);
When you're paranoid and use -Werror -Wredundant-decls you'll notice a duplicate declaration in pyerror.h. Bug filed.
The module initialisation is a lot easier then in the official docs. You seem perfectly fine without module state, so all you need to fill out in your struct PyModuleDef is m_base, m_name and m_size. Of course m_doc and m_methods are pretty useful too, but not strictly required. Copy the tutorial here.
And if you use PyMODINIT_FUNC to declare it all you need to #ifdef is PyInitmymodule(void), PyModule_Create() and the return value.

Generating source files in setup.py

2009-01-09T12:24:00.004Z

I have a Python extension module written in C and one of the weirder things it needs to do is have a number of module constants with their values which are defined (e.g. #define PREFIX_FOO 0x01 etc.) in an external C header file. All the defined names in that header file start with a common prefix so it's easy enough to write a python function that will read the file and spit out the correct C source code that enables me to expose these in my python module. The tricky part however is where to hook this up in the setup.py script.

At first I tried to extend the distutils.command.build_ext.build_ext class to generate this file. That doesn't work however as the distribution is not very happy about having a file listed as required (Extension(sources=['generated_file.c', ...], ...)) which isn't actually there at the time the distutils.dist.Distribution instance is created.

So the two (AFAIK) remaining options are to subclass distutils.dist.Distribution and pass into the setup method: setup(distclass=MyDistribution, ...). Or secondly, don't add the generated source file to the list of required files and created in the extended build_ext command.

For now I've gone for the last option as it seems the more appropriate place to do things (I've overwritten the .initialize_options() method). But I wonder if other more elegant solutions exist?

import skynet

2008-12-24T09:09:00.001Z

So who's going to make the skynet module?

Finding memory leaks in Python extension modules

2008-12-13T12:02:00.002Z

Valgrind is amazing, even if you've never used it before it's basic usage is really simple:

$ valgrind --tool=memcheck --leak-check=full /usr/bin/python ./test.py

This is enough to point you at the places in your extension module where you allocated stuff that didn't get freed later. This is a massive timesaver with looking over the entire source file again to find out where you made your mistakes.

I must admit that the extension module in question uses malloc(3) and free(3) directly instead of allocating on the Python heap using PyMem_Malloc() and PyMem_Free(), so I don't know if that would make it harder to find the leaks. I can imagine that in that case the "blocks definitely lost" list might point to somewhere in Python's source files instead of your own source files, but I don't know.

gcc, LD_RUN_PATH & Solaris

2008-12-11T00:48:00.002Z

I actually started writing a long post about how linking agains shared libraries works and what and how ld.so does, but that seemed to get very long. So here the short version that assumes pre-existing knowledge.

If you want an RPATH in your DSOs (dynamic shared objects: executables and shared libraries) you can pass in -R/-rpath to the linker (the runtime_library_dir keyword argument to distutils's Extension class). This is not always very practical though, e.g. when building Python it would be a major pain to modify all the makefiles and setup.py. Error prone too.

So the linkers (both GNU and Solaris) also accept an environment variable: LD_RUN_PATH. When that is set it is used to populate the RPATH field, but only when no explicit -R/-rpath is specified. So far so good.

On Solaris however your gcc will usually not be installed in the root system, rather in /usr/sfw (sun supplied) or /opt/csw/gccX (opencsw). So the gcc libs will also not be in the default library search path. But gcc is nice and helps you out, it will implicitly add a -R option to ld pointing to it's own library directory (for libgcc). Now I'm not sure how nice exactly this is since it screws your LD_RUN_PATH environment variable over and you actually need to run gcc traced to see this happening, have fun finding that! It would be nice if gcc would extend the environment variable if you had it set but where using no -R flags instead. Oh well, at least now you know.

Mocking away the .init() method

2008-12-06T22:47:00.003Z

I wanted to test a method on a class that didn't really depend on a lot of other stuff off the class. Or rather what it did depend on I had already turned into Mock objects with appropriate .return_values and asserting with .called etc. Problem was that the .__init__() method of the object invoked about half the application framework (option parsing, configuration file loading, setting up logging etc.). Firstly I don't really feel like testing all of that (hey, these are called unittests after all and those fuctionalities have their own!) and secondly then I had to worry about way too much, quite a lot to setup.

That's how the whacky idea of replacing the .__init__() method with a mock occurred to me:

class TestSomething(object):
    @mock.patch('module.Klass.__init__',
                mock.Mock(return_value=None))
    def test_method(self):
        inst = module.Klass()
        inst.other_method = mock.Mock()
        inst._Klass__log = mock.Mock()
        # more of this
        inst.method()
        assert inst.other_method.called

The ugly side effects of deciding to mock away the .__init__() method like this is that I have to create mocks for more internal stuff. The one shown for exampls is normally provided by self.__log = logging.getLogger('foo').

I must admit that I'm still trying to find my way in how to use the mock module effectively and hence I'm not really sure how sane this approach is. One of my objections with this is that not only am I meddling with clearly hidden attributes of the class, but I also have to do this again and again for each test method. So the next revision (I'm using py.test as testing framework here btw):

class TestSomething(object):
    @classmethod
    def setup_class(cls):
        cls._original_init_method = module.Klass.__init__
        module.Klass.__init__ = mock.Mock(return_value=None)

    @classmethod
    def teardown_class(cls):
        module.Klass.__init__ = cls._original_init_method

    def setup_method(self, method):
        self.inst = module.Klass()
        self.inst._Klass__log = mock.Mock()
        # more of this

    def test_method(self):
        self.inst.other_method = mock.Mock()
        self.app.method()
        assert self.inst.other_method.called

This is actually workable and I'm testing what I want to test in a pretty isolated way. I'm still wondering whether I've gone insane or not tough. Is it reasonable to replace .__init__() by mock objects? Have other people done this?

Preserving modification times in subversion

2008-11-26T17:41:00.002Z

This is an oft requested feature apparently. But who know if and when it will be there. Usually things get done when someone has an itch to scratch, but given that no one has done this yet -but it's been requested for ages- it seems that it must either be really difficult or work arounds are easier. Turns out that in my case a work around is a lot easier.

I need to build various bits of software on various platforms and they can have local modifications. In the case of Python for example we strip out the bit where it looks up the modules search path in the registry on Windows (instead of clobbering a hopefully unique string in the resulting binary by a hopefully non-existing registry key like py2exe does IIRC - I'm surprised that hasn't blow up in anyones face yet but nevermind). Anyway, the point of this is that the natural format to store all of this in was in unpacked form in the revision control system, subversion in our case, so we can diff and merge as we like. Problem with this is that when we check out the source on a build host all timestamps are lost and for some projects this messes up the makefile logic and things break down horribly and randomly on the build hosts. I got tired of finding this out every time and adding yet another random touch to the build scripts so was looking for something better.

Turns out that this is actually suprisingly easy to do. I just store the timestamps of the unpacked tarball in a file, and call touch on all files first thing in our build script. These shell functions are not bomb-proof yet (it won't cope with spaces in filenames e.g.) but do the job for now:

create() {
    if [ -f .mtimes ]; then
        echo "E: .mtimes exists already!" >&2
        exit 1
    fi

    for file in $(find . -print); do
        mtime=$(stat --format=%y "$file")
        echo $file $mtime >> .mtimes
    done
}


restore() {
    if [ ! -f .mtimes ]; then
        echo "E: No .mtimes file found!" >&2
        exit 1
    fi

    while read file mtime; do
        # This would be the simple GNU option, POSIX however...
        #touch --date="$mtime" $file
        CCYY=$(echo $mtime | cut -d- -f1)
        MM=$(echo $mtime | cut -d- -f2)
        DD=$(echo $mtime | cut -d- -f3 | cut -d' ' -f1)
        hh=$(echo $mtime | cut -d' ' -f2 | cut -d: -f1)
        mm=$(echo $mtime | cut -d' ' -f2 | cut -d: -f2)
        SS=$(echo $mtime | cut -d' ' -f2 | cut -d: -f3 | cut -d. -f1)
        touch -t $CCYY$MM$DD$hh$mm.$SS $file
    done < .mtimes
}

Note that stat is a GNU tool in the coreutils package, so create() will only work on a GNU-based system. restore() however should work on all POSIX compliant systems (and so far it seems to do so).

Only thing is that touching each and every file is a rather slow process... And before you ask, I actually do the timestamp conversion at creation time as that happens less often, but this seems clearer.

Generative Tests

2008-09-16T20:48:00.002+01:00

Both nose and py.test allow you to write generative tests. These are basically generators that yield the actual test functions and their arguments. The test runners will then test all the yielded functions with their arguments.

However I'm left wondering why this is better then just iterating in the test function and doing many asserts? Let me demonstrate with the py.test example:

def test_generative():
    for x in (42,17,49):
        yield check, x

def check(arg):
    assert arg % 7 == 0   # second generated tests fails!

Why is that code better then:

def test_something():
    for x in (42,17,49):
        assert x % 7 == 0

I know that it could possibly tell you slightly more, but in general I keep testing until all of my tests succeed so I don't really mind if one test includes more then one assert statement. I'll just keep fixing things till the test passes.

What is the motivation for generative tests?

As an aside (and the inspiration for this post thanks to Holger Krekel talking about py.test), PyCon UK was great. I was considering a post titled "Resolver Systems Ltd is Evil since they provided the Sunday morning hangovers, but thought that was a bit too sensationalist.

Solaris equivalent of dpkg -S

2008-08-19T12:08:00.003+01:00

In the past I have spent quite some time trying to figure out what package did install a certain file on Solaris. What I wanted was really just the equivalent of dpkg -S /path/to/file. But those searches where not very fruitfull, even talking to more experienced Solaris admins didn't help, they could only tell me how to find out what files where in a certain package.

Today however, mostly by accident (I was trying to remember how to list the files in a package), I finally found it!

neptune:~# pkgchk -l -p /usr/local/bin/wget 
Pathname: /usr/local/bin/wget
Type: regular file
Expected mode: 0755
Expected owner: bin
Expected group: bin
Expected file size (bytes): 392572
Expected sum(1) of contents: 32029
Expected last modification: Nov 25 10:57:55 2006
Referenced by the following packages:
        SMCwget        
Current status: installed

neptune:~#

Wonderful

Bluetooth on a Toshiba Tecra A9-127

2008-06-14T23:51:00.005+01:00

I got a Toshiba Tecra A9-127, listening to the model number of PTS52E-06002LEN as written on the back, from work to replace my dying old HP Compaq nx9030. As my laptop OS of choice is Ubuntu, currently in it's Hardy release, it's not completely coincidence that the hardware is almost all Intel based since that's what Matthew Garrett recommends. And indeed, it all works effortlessly! Apart from bluetooth.

For bluetooth you need a tool called toshset. Once you have that you can enable the internal bluetooth device:

flub@signy:~$ sudo toshset -bluetooth on
bluetooth: attached
flub@signy:~$

And all of a sudden you'll have an hci device, probably hci0, check it with hciconfig -a if you fancy. Magic! It's just like plugging in a USB dongle...

Only toshset is not available for the amd64 flavour of hardy, only in the i386 version. No panic though, Debian has an up to date package (Ubuntu intrepid also has the right 1.73 version but doesn't build it for amd64 yet - bug filled).

$ dget http://the.earth.li/debian/pool/main/t/toshset/toshset_1.73-2.dsc
$ dpkg-source -x toshset_1.73-2.dsc

Don't quite rejoice yet, Debian seems to have changed from the pciutils-dev package to libpci-dev. So go and edit debian/control to build depend on pciutils-dev again. Then just build the package, install and enjoy.

Hopefully someone will now spend less time then me figuring this out...

Ripping videos from DVD

2008-05-10T19:24:00.003+01:00

Physical discs are a nuisance, I really just want to play what I want to watch in the room I want to watch it just streaming it over the wireless. This actually works wonderfully well. Unfortunately just copying the file structure of a DVD works and gives you DVD-quality video, but the size is huge and streaming this over the wireless tends to create some trouble (not to mention that every byte must also be encrypted/decrypted for ssh so this is getting CPU intensive too, ssh offloading onto hardware would be so cool). Hence the need to "rip" the DVD and encode it to some smaller format arises.

After spending a while looking at various options I finally found the great thoggen.

Only problem left is how much to compress. After some very un-scientific tests (Google failed to find me any nice studies/graphs!) I decided on a quality of 35 and no resizing. But if anyone knows of a better study on what settings to prefer it would be greatly appreciated! It would be great to see quality vs file size vs frame size for different types of video. Ideally with a subjective "human quality" level too so you'd know what still looks good.

Discovery of the day

2008-05-09T11:31:00.001+01:00

>>> def f(a, b):
...     print 'a', a, 'b', b
... 
>>> f(1, b=2)
a 1 b 2
>>> f(a=1, b=2)
a 1 b 2
>>> f(b=2, a=1)
a 1 b 2
>>>

Now I'm debating if I should really use that in production code...

Time to read standards

2008-04-17T08:45:00.002+01:00

Sometimes I like quotes out of context...

Anyway, I don't think it really is an ambiguity in practice -- only in the minds of those that have too much time to read standards documents.
-- Greg Ewing (on distutils-sig)

@x.setter syntax in Python 2.5

2008-04-11T20:49:00.002+01:00

The new property.setter syntax in Python 2.6 and 3.0 looks very clean to me. So I couldn't wait and wanted it in Python 2.5. With the help of comp.lang.python I finally got there (thanks especially to Arnaud Delobelle):

_property = property

class property(property):
    def __init__(self, fget, *args, **kwargs):
        self.__doc__ = fget.__doc__
        super(property, self).__init__(fget, *args, **kwargs)

    def setter(self, fset):
        cls_ns = sys._getframe(1).f_locals
        for k, v in cls_ns.iteritems():
            if v == self:
                propname = k
                break
        cls_ns[propname] = property(self.fget, fset,
                                    self.fdel, self.__doc__)
        return cls_ns[propname]

The __init__() wrapper is needed to get __doc__ set properly (this surprised me). The whole cls_ns stuff and the loop are required since the properties are defined in C and their fset attribute is read-only. Which is why the entire property needs to be replaced. The implementation of deleter() can now be regarded as an exercise to the reader...

(O)OXML and ISO voting processes

2008-04-11T20:42:00.002+01:00

Many have recently complained about ISO's voting processes, mainly how they need to be revised as currently it seems it's possible to buy yourself a standard given enough lobbyist and money.

But part of this is ISO's trust in ECMA. ISO allows ECMA to submit standards for the fast-track process because it trusts it to approve good standards. If OXML (as it seems to be called now it's approved, formerly OOXML) was indeed such a bad standard (which I don't doubt personally) then ISO should maybe review it's relationship with ECMA too?

This is only a tiny part of the picture obviously, but one I haven't seem mentioned elsewhere.

Shell history

2008-04-10T13:57:00.002+01:00

hehe

flub@signy:~$ history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn |head
79 nosetests
74 ls
46 cd
36 ssh
28 man
22 apt-cache
19 vi
19 ack
17 svn
17 sudo
flub@signy:~$

It must be noted that most of my editing happens in emacs, but that only gets started a few times a day and then stays there (oh and emacs is started from a shortcut icon, not the shell).

nose and ipython

2008-04-06T13:36:00.002+01:00

I wish there was a simple way to integrate nose and ipython. Something like a command line option not to catch exceptions would be sufficient I think, so that you could do:


In [1]: %run tests/test_module.py --do-not-catch-exceptions

Obviously you'd want a shorter option switch...

Seems like Test.run() in case.py is where this happens, but I tried changing that with no success.

And to be really useful I'd want the default behaviour of test selection in a module where if __name__ == '__main__': nose.main() is used to be just that module. But maybe that's already supported and I'm just not finding it.

Last random thought: Maybe if nose's -d option did expand dotted names I wouldn't be wishing for any of this. Who knows.

One s3fs to rule them all?

2008-04-03T12:38:00.003+01:00

Sometimes I wish we could fast forward 6 or 12 months or so. Hopefully by then there will be just one s3fs that is mature and maintained. Right now it's hard to tell which one will turn out the best.

Interstingly, of the ones that look like they could have potential (trying to word it carefully here) two are written in Python (all are using fuse).

GPL and python modules

2008-03-28T23:42:00.003Z

I should probably google this, but can't think of good keywords. So instead this is a "dear lazy web" post.

Is importing a python module considered linking for the GPL? I.e. are you allowed to import a python module into an script or module that has a non-GPL-comptible license?

Thanks!

Cheating the KAME turtule

2008-03-27T21:35:00.002Z

This is shocking. Even though I have not yet used IPv6 I can see the the KAME turtle dance! I won't give you a direct link but SixXS privde an IPv6->IPv4 service as well as an IPv4->IPv6 service. Feels bad having used the 4->6 bit...

OmniORB 4.1.1 in lenny!

2008-03-23T18:54:00.000Z

Something that happened about a week ago I think. But omniORB 4.1.1 finally make it into Debian testing aka lenny! This means that python-omniorb 3.1, which has been waiting for a while now, also make it into lenny!

It has been stalled for ages since we had problems with the ARM port not managing to compile omniORB 4.1.X. Thanks to Thomas Girard for his work on this.

I've already committed the update to 4.1.2 to the svn repo, omniORBpy 3.2 will follow soon...

Compiling libtool examples

2008-03-23T18:23:00.002Z

This is more a reminder to myself, but here's how to compile the libtool examples:

$ cd demo
$ aclocal
$ libtoolize --automake
$ automake --add-missing
$ autoconf
$ ./configure
$ make

It's really how to bootstrap any GNU autotools using application I suppose.