devork

E pur si muove

Selectable queue

Saturday, May 29, 2010

Sometimes you'd want to use something like select.select() on Queues. If you do some searching for this it turns out that this question has been answered for multiprocessing Queues where you can simply use the real select on the underlying socket (IIRC), but for a good old fashioned Queue you're stuck.

Now it's easy to argue that this isn't that high a need, when I wanted this a while ago it turned out to be surprisingly simple to re-structure the design a little so that I no longer desired a selectable queue. But it's still something that hung around the back of my mind for a while, so I've kept thinking about it. My conclusion for now (which I haven't bothered implementing) is that simply cloning the normal queues but replacing the not_empty and not_full Conditions by Events gives you selectable queues. It changes the semantics slightly, but that doesn't seem harmful. This obviously isn't enough, so the second change to the queue is that you should be allowed to pass in the events to use. And now that you can share the not_full event between two queues you can simply wait on this event and you have your select.

Next time I want this I might actually implement this idea rather then re-design so that I don't want selectable queues anymore.

weakref and circular references: should I really care?

Wednesday, May 12, 2010

While Python has a garbage collector pretty much whenever circular references are touched upon it is advised to use weak references or otherwise break the cycle. But should we really care? I'd like not to, it seem like something the platfrom (python vm) should just provide for me. Are all those mentions really just for the cases where you can't (or don't want to, e.g. embedded) use the normal garbage collector?

Update: By some strange brain failure I seemed to have written "imports" rather then "references" in the title originally. They are obviously a bad thing.

python-prctl

Saturday, May 08, 2010

There is sometimes a need to set the process name from within python, this would allow you to use something like "pkill myapp" rather then the process name of your application just being yet another "python". Bugs (which I'm now failing to find in Python's tracker) have been filed about this and many wannabe implementations made, all of them seemed to have many problems however. Aiming for UNIX cross-compatibility they all messed up and end up trying to make the abstraction everywhere, and ususally the code was less then beautiful

But I've just discovered the python-prctl module by Dennis Kaarsemaker which does something far more sensible: rather then trying to be cross-platform it just wraps the Linux prctl system call (as well as libcap). The code looks well written and the API, while a little bit overloaded on the get-set names, seems nice. It even includes what is probably the most sensible implementation of clobbering argv that I've ever seen (but don't use that, no normal person should ever clobber argv!).

If someone writes a nice module like this to cater for the MacOSX guys, the only other system I know of has a system call to set the process name, then I may never have to worry about getting someting like this into PSI at some distant point in the future. (And speaking about PSI, yes the windows port is still slowly under way. I"m just busy with lots of other things at the same time.)

Storm and sqlite locking

Saturday, May 08, 2010

The Storm ORM struggles with sqlite3's transaction behaviour as they explain in their source code. Looking at the implementation of .raw_execute() the side effect of their solution to this is that they start an explicit transaction on every statement that gets executed. Including SELECT.

This, in turn, sucks big time. If you look at sqlite's locking behviour you will find that it should be possible to read from the database using concurrent connections (i.e. from multiple processes and/or threads at the same time). However since storm explicitly starts that transaction for a select it means the connection doing the read now holds a SHARED lock until you end the transaction (by doing a commit or rollback). But since it's holding this shared lock, for no good reason, it means no other connection can acquire the EXCLUSIVE lock at the same time.

The upshot of this seems to be that you need to call .commit() even after just reading the database, thus ensuring you let go of the shared lock. Can't say I like that.

Subscribe to: Posts (Atom)