devork

E pur si muove

__slots__: worth the trouble?

Wednesday, May 30, 2007

I find __slots__ annoying. They seem to have a lot of rules about their use and make things like a borgs rather awkward to use (I even prefer to use a singleton instead of a borg in new-style classes because of this).

And all it does is safe some space in memory (AFAIK). Make me wonder if it's worth it. Google claims there are about 2k hits for __slots__. Compare to e.g. about 300k hits when doing the same search for __init__.

Maybe I'm just overreacting though. Maybe having all class data in one place is not that important. Or maybe there are just massive gains in using __slots__ for some people that do make it worth it. But I'm not one of them so hence my rant.

IPC in the stdlib(?)

Thursday, May 17, 2007

Many interesting comments on my last musings about IPC. Each and every one of them look very nice and have their application area, go and have a look! These found a weak spot in my heart though:

  • Candygram is Erlang style message passing. Very stunning, but to my liking a little bit too much Erlang-like. Nevertheless I was impressed.
  • The delegate module was probably the closest to what I was describing. And I'm surprised it's not better known.
  • Lastly I can't keep my mind off pylinda. From the moment I started to read about it I fell in love with it. It is Erlang-like enough to satisfy me and also has a huge potential even to talk to other languages. I'm not sure yet how it's API looks as I'd want to do things like running the server as part of the main process (and it being a private server as well then) etc. It got me excited allright.

All this is well though, but I'm not sure I made my main point very clear last time (and fear losing it here in the noise too, I must be a bad writer). Namely, non of this is in the stdlib (yes, xmlrpc is).

I do think there is a need for a good and yet simple pythonesque IPC module in the stdlib. When building cluster or grid systems it seems reasonable to me to stray outside of the stdlib (for now at least). But for simple SMP a module should be making it's way into the stdlib by now IMHO.

Given by what I've seen so far -but beware, I haven't used any of these yet- delegate seems like the best suited candidate. It may need some updating, I can imagine it benefitting from the subprocess module for example, but I certainly think something like it should live in the stdlib.

I wonder what the chances of that happening are.

IPC and the GIL

Wednesday, May 09, 2007

As recently described excelently, threads are ugly beasts, waiting to get you when you let your guard down (and eventually you will). Yes, that means that I should really get my head round the asynchat module and stop using ThreadingMixIns with the SocketServer, but that's not the point of this post.

Inter process communication, aka IPC, is way safer and scalable when you want to distribute the work between many processors. But what bothers me is the complete lack of a nice support for this in the stdlib, while there is an excellent threading module.

So what options are there?

You could stay inside the stdlib and construct something out of pickle and sockets, but that isn't amazing as you'll have to write a lot of boiler plate code that will have it's own bugs. Leaving the stdlib you could probably replace pickle with json, but appart from being able to talk with non-python code you're not really better off.

There is of course the excellent omniORBpy if you grok CORBA. But the python mapping of CORBA was written by C++ programmers, so despite being the best CORBA mapping available it is still not the most lovely thing to work with as python developer. And CORBA is a rather heavy weight approach that is far from applicable in all situations. XML-RPC is another internet-aware IPC option that even has support in the stdlib, but once more the overhead when you're just on a local machine and want to use 2 CPUs is silly. Twisted must surely have some solution too, but I don't really know that and again I think it will be rather heavy-weight. Lastly I feel obliged to mention SOAP in this paragraph too, but can't help to shrudder when thinking about it.

I guess what I'm looking for is some simple, light weight and scalable module that does something similar to how Erlang passes messages around. Maybe all I'm asking for is a module that ties up subprocess and pickle and lives under the IPC section in the stdlib. But it would be nice if it could also transparantly stream the data accross a socket so you can also run on multiple hosts if you want (maybe in a separate module though). Having a module like that around would obsolete the need for boilerplate code as well as establishing some sort of "best practice" IPC.

Anyways, I think this is more like me wondering what people use or would want to have for their IPC in python. Anyone?

More roundup love

Tuesday, May 01, 2007

As I've said before, roundup is an amazing issue tracker. Last time I wrote I had just implemented our internal bug tracker with all the weird requirements of a few developers. By now the incredibly flexible system, thanks to python, has been moulded into our complete project management tracker.

This second instance tracks our development effort from customer wishes through to specifications, development and passing of acceptance criterias. All with the help of a few (three actually) IssueClasses and some funky relationships between them with the help of python functions as detectors. The last part of the equation is our good old wiki (moinmoin if you where wondering).

PS: No it didn't take as long as these two posts about roundup are appart. For one I did lots of other stuff in between. And secondly I've been meaning to tell people how flexible roundup is for a while now.

Campaigning for -delete

Tuesday, May 01, 2007

Whenever find(1) is used to delete some files there are always a few variants that can be seen. They are generally combinations of -exec and -print piped to xargs. While demonstrating the increadible power of UNIX shells and pipes etc they don't appear very practical.

Most implementations struggle with embedded spaces, newlines, quotes etc. So even more exciting things get produced to cope with all these weird things. So why not simply use -delete at the end of the desired expression? Is it because it is a GNU extension? I swear that most of the expressions that actually try to cope with weird filenames depend on GNU extensions anyway. Not to mention that most of the scripts know they will have GNU find and don't behave scrictly POSIX compliant anyway.

So why is it that in no example -delete is used? There's no point as far as I can tell.

In light of that this post is my campain for the more frequent use of -delete in find(1) expressions. I'm sure you won't care. Neither will I; by tomorrow I'll have forgotten about this campain. But at least I'll be using -delete myself.

Subscribe to: Posts (Atom)