Knowing almost nothing about frontend development or GIS I wanted to be able to display something on a map on my own computer (desktop). Knowing a fair bit of Rust but not having a clue about anything else involved I decided the easiest way to do this was probably to …
Blog Archives
-
-
Using Modern Linux Sockets
Introduction
On the surface of it the Linux socket API is pretty straight forward. However once you start to care about performance things get a bit more complicated, and documentation a bit more sparse.
Because I thought it'd be fun to explore this a bit more I made a simple …
-
Experiments with an Email Protocol
Background
The primary way I use email is through notmuch and notmuch-emacs on my laptop, which reads email from a local maildir. Sending mail is done using the venerable sendmail(8) command, which is backed by nullmailer to send it out via SMTP to the relay server of my email …
-
Encrypted root on Debian with keyfile/keyscript
I've recently set up another laptop with whith whole-disk encryption, including /boot. This is all fine --the debian-installer almost gets it right, you just need to edit /etc/default/grub to add GRUB_ENABLE_CRYPTODISK=y and then re-run grub-install on the target-- but once you get this working you end up …
-
A Container is an Erlang Process
This post is a response to A Container Is A Function Call by Glyph. It is a good article and worth your time reading, and you might want to read it to follow here. On twitter I asserted the article recommends building a monolith while Glyph countered "On the contrary …
-
py.test sprint in Freiburg
Testing is a really important part of Python development and picking the testing tool of choice is no light decision. Quite a few years ago I eventually decided py.test would be the best tool for this, a choice I have never regretted but has rather been re-enforced ever since …
-
Pylint and dynamically populated packages
Python links the module namespace directly to the layout of the source locations on the filesystem. And this is mostly fine, certainly for applications. For libraries sometimes one might want to control the toplevel namespace or API more tightly. This also is mostly fine as one can just use private …
-
New pytest-timeout release
At long last I have updated my pytest-timeout plugin. pytest-timeout is a plugin to py.test which will interrupt tests which are taking longer then a set time and dump the stack traces of all threads. This was initially developed in order to debug some some tests which would occasionally …
-
Designing binary/text APIs in a polygot py2/py3 world
The general advice for handling text in an application is to use a so called unicode sandwich: that is decode bytes to unicode (text) as soon as receiving it, have everything internally handle unicode and then right at the boundary encode it back to bytes. Typically the boundaries where the …
-
Don't be scared of copyright
It appears there is some arguments against putting copyright statements on the top of a file in free software or open source projects. Over at opensource.com Rich Bowen argues that it is counter productive and not in the community spirit (in this case talking about OpenStack). It seems to …
-
Setting up a Python development environment on Solaris 11
Solaris is one of those things which tends to be a bit un-loved, it being baught by Oracle probably didn't help either. But regardless they've made a rather nice and modern OS out of it and it is used in a fair few places so you may need to support …
-
Inspecting un-imported modules using ast
or
Skipping modules in py.test using marks but no importing
As I've said before, py.test is my favourite testing tool. One if it's features is that it allows you to mark tests with arbitrary "marks", built in ones are e.g. skipif, xfail etc. But py.test's extension …
-
Small Emacs tweaks impoving my Python coding
Today I've spent a few minutes tweaking Emacs a little. The result is very simple yet makes a decent impact on usage.
Firstly I remembered using the c-subword-mode a long time ago, I couldn't believe I never used that in Python. Turns out there is a more genericly named subword-mode …
-
Using __getattr__ and property
Today I wasted a lot of time trying to figure out why a class using both a __getattr__ and a property mysteriously failed. The short version is: Make sure you don't raise an AttributeError in the property.fget()
The start point of this horrible voyage was a class which looked …
-
Synchronising eventlets and threads
-
Creating subprocesses in new contracts on Solaris 10
Solaris 10 introduced "contracts" for processes. You can read all about it in the contract(4) manpage but simply put it's a grouping of processes under another ID, and you can "monitor" these groups, e.g. be notified when a process in a group coredumps etc. This is actually one …
-
re.search() faster then re.match()
This is a counter-intuitive discovery, in IPython:
In [18]: expr = re.compile('foo') In [19]: %timeit expr.search('foobar') 1000000 loops, best of 3: 453 ns per loop In [20]: %timeit expr.match('foobar') 1000000 loops, best of 3: 638 ns per loop
So now I'm left wondering why .match …
-
Storm and SQLite in-memory databases
When using an SQLite in-memory databases in storm the different stores created from the same database are not modifying the same SQLite database. E.g.
db = storm.locals.create_database('sqlite:') store1 = storm.locals.Store(db) store2 = storm.locals.Store(db)
Here store1 and store2 will not refer to the same …
-
Finding the linux thread ID from within python using ctypes
So I've got a multi-threaded application and suddenly I notice there's one thread running away and using all CPU. Not good, probably a loop gone wrong. But where? One way to find this is revert history in the VCS and keep trying it out till you find the bad commit …
-
Return inside with statement
Somehow my brain seems to think there's a reason not to return inside a with statement, so rather then doing this:
def foo(): with ctx_manager: return bar()
I always do:
def foo(): with ctx_manager: result = bar() return result
No idea why nor where I think to have heard/read this …
-
Templating engine in python stdlib?
I am a great proponent of the python standard library, I love having lots of tools in there and hate having to resort to thirdparty libs. This is why I was wondering if there could be a real templating engine in the stdlib some day? I'm not a great user …
-
Using Debian source format 3.0 (quilt) and svn-buildpackage
Searching the svn-buildpackage manpage for the 3.0 (quilt) format I thought that it wasn't able to apply the patches in debian/patches during build time. Instead I was doing a horrible dance which looked something like svn-buildpackage --svn-export; cd ../build-area/...; debuild. Turns out I was completely wrong.
svn-buildpackage doesn't …
-
Europython, threading and virtualenv
I use threads
correctly
I do not use virtualenv
and dont't want to
Just needed to get that out of my system after europython, now mock me.
PS: I should probably have done this as a lightening talk but that occurred to me too late.
-
py.test test generators and cached setup
Recently I've been enjoying py.test's test function arguments, it takes a little getting used too but soon you find that it's quite likely a better way then the xUnit-style of setup/teardown. One slightly more advanced usage was using cached setup together with test generators however. While not …
-
Selectable queue
Sometimes you'd want to use something like select.select() on Queues. If you do some searching for this it turns out that this question has been answered for multiprocessing Queues where you can simply use the real select on the underlying socket (IIRC), but for a good old fashioned Queue …
-
weakref and circular references: should I really care?
While Python has a garbage collector pretty much whenever circular references are touched upon it is advised to use weak references or otherwise break the cycle. But should we really care? I'd like not to, it seem like something the platfrom (python vm) should just provide for me. Are all …
-
python-prctl
There is sometimes a need to set the process name from within python, this would allow you to use something like pkill myapp rather then the process name of your application just being yet another "python". Bugs (which I'm now failing to find in Python's tracker) have been filed about …
-
Storm and sqlite locking
The Storm ORM struggles with sqlite3's <http://docs.python.org/library/sqlite3.html> transaction behaviour as they explain in their source code. Looking at the implementation of .raw_execute() the side effect of their solution to this is that they start an explicit transaction on every statement that gets executed. Including …
-
Python optimisation has surprising side effects
Here's something that surprised me:
a = None def f(): b = a return b def g(): b = a a = 'foo' return b
While f() is perfectly fine, g() raises an UnboundLocalError. This is because Python optimises access to local variables using the LOAD_FAST/STORE_FAST opcode, you can easily see why this …
-
Judging performance of python code
Recently I"ve been messing around with python bytecode, as a result I now know approximately how the python virtual virtual machine works at a bytecode level. I've found this quite interesting but other then satisfying my own curiosity had no benefit from this knowledge. Until today I was wondering …
-
Hacking mock: Mock.assert_api(...)
Mock is a great module to use in testing, I use it pretty much all the time. But one thing I have nerver felt great about is the syntax of it's call_args (and call_args_list): it is a 2-tuple of the positional arguments and the keyword arguments, e.g. (('arg1', 'arg2' …
-
Using lib2to3 in setup.py
It seems that many people think you need distribute if you want to run 2to3 automatically in your setup.py. But personally I don't like setuptools (aka distribute) and hence don't like forcing this on users. No worries since plain old distutils supports this as well, but it simply appears …
-
Discovering basestring
This may seem simple and trivial, but today I discovered the basestring type in Python. It is essentially a base type for str and unicode, which comes in handy when writing isinstance(foo, basestring) in test code for example.
Strangely, despide PEP 237 mentioning the equivalent for int and long …
-
Registering callbacks: to quack like a duck or tell which bit will quack the duck way?
When you have someplace where you register a set of callbacks you have a few options:
- Use an interface-style API. I.e. you have a Mixin style class defining the methods expected to be present, probably just raising NotImplementedError. You then just pass in the entire object to the .register …
-
Pointer arithmetic in C
Pointer arithmetic in C is great. The only downside is that it's only defined when pointing to an array (or an item just after the array). That is one giant downside.
Allocate a chunk of memory and you can't use pointer arithmetic. So if you are building a linked list …
-
The master plan
Catherine Devlin describes it very nicely:
I finally understand Al-Qaeda's master plan, and it's freaking brilliant. [...] I'm just surprised that we're choosing to participate in the plan. I thought we were on opposite sides?
-
Terminology
Why was it ever considered desirable to call a directory containing a __init__.py file a "package" rather then just "module". They are after all simply "modules containing other modules". It's not like that solved some sort of problem was it? But, as sad as that sometimes might be, we …
-
Installing Debian on a Sun Fire X2200 M2
Most important thing first: the eLOM console lives on ttyS1 (9600n8). If that gets you going you do not need to read any more.
This works pretty easily, but there's isn't much information about how this works if you don't like using a keyboard and a screen (having these things …
-
Decorators specific to a class
My gut tells me it's horribly wrong but I am failing to formulate a decent argument, let me show an example what I mean (somewhat contrived):
def deco(f): def wrapper(self, *args, **kw): with self.lock: f(self, *args, **kw) class Foo: def __init__(self): self.lock = threading.Lock …
-
Calling COM methods in C
-
Setting descriptors on modules
This counts for one of the more crazy things I'd like to do with Python: insert an instance of a descriptor object into the class dict of a module.
While you can get hold of the module type class by using the __class__ attribute of any module or use types …
-
Skipping slow test by default in py.test
If you got a test suite that runs some very slow tests it might be troublesome to run those all the time. There is of course the "-k" option to py.test which allows you to selectively enable only a few tests. But what I really wanted was a way …
-
setuptools vs distribute
In case you heard of both setuptools and distribute: distribute fully replaces setuptools. Just use distribute. Setuptools is “maintained” (for various historically dubious values of “maintain”) by one person (whom all should applaud for creating the darn thing in the first place, btw!). Distribute is maintained by …
-
Dictionary entry
innovative company
noun 1 a company which can't afford patent lawyers *2 one of the lucky handful companies who are innovative and have money for patent lawyers *HELP Not to be confused with most companies lobying in patent litigation.
PS: It's also possible to have companies that can't afford patent …
-
How to drive on a motorway
If you are driving on a motorway where I am driving too, here are some rules you should follow. I think you should even follow them when I'm not around.
- Be sensible, this one overrules all the others
- Keep left unless overtaking
- KEEP LEFT UNLESS OVERTAKING
- KEEP LEFT UNLESS OVERTAKING …
-
How to make a crosslinked RS-232 cable
Because it's easier then buying one
When connecting two computes together (e.g. your laptop to a server or router) to get access to a console you need to use a crosslinked RS-232 cable, usually with two female DE-9 connectors these days. This cable is more commonly known as a …
-
On using .__getstate__()
Do not, I repeat, do not define .__getstate__() idly.
Thank you very much
-
Tuple unpacking goodness
Todays pleasant surprise:
>>> a = {'a': (0, 1), 'b': (2, 3)} >>> for k, (v1, v2) in a.iteritems(): ... print k, v1, v2 ... a 0 1 b 2 3 >>>
Nice!
-
The new faces of Europe
I can't help but agree.
-
Finding memory leaks in extension modules
After reading Ned Batchelder's post on his experience hunting a memory leak (which turned out to be a reference counting error) it occurred to me that even tough I have a script to check memory usage I should also really be checking reference counts with sys.gettotalrefcount(). And indeed, after …
-
New Python System Information release!
I've just released a new version of PSI! PSI is a cross-platform Python package providing real-time access to processes and other miscellaneous system information such as architecture, boottime and filesystems. Among the highlights of this release are:
- Improved handling of time
- We now have our own object to represent time …
-
Synchronous classes in Python
What I'd like to build is an object that when doing anything with it would first acquire a lock and release it when finished. It's a pattern I use fairly-regularly and I am getting bored of always manually defining a lock next to the other object and manually acquiring and …
-
Python modules and the GPL: I still don't get it
I've never understood if you can use GPL python modules (or packages) in GPL-incompatibly licensed code. Today I re-read the GPL looking for this and am tempted to think yes, you can. The GPL says:
- You may copy and distribute verbatim copies of the Program's source code as you receive …
-
Delny 0.4.1
A little while ago I released a new version of Delny (my python wrapper around Qhull for Delaunay triangulations), the main purpose to use numpy instead of numeric. Impressively enough people actually seemed to care and I got a few bug reports and hints for improvements.
So I just released …
-
No permission to see Ubuntu bugs?
So I'm looking at the release notes for Ubuntu 9.10 and am interested in the hibernation issue. Naturally I follow the link to the bug report.
First I need to log in to launchpad. That's weird, since when do I need to log in simply to view bugs? But …
-
Cross platform shell scripts
At work we have an rather elaborate collection of shell scripts that builds our softweare and all it's dependencies on about 7 UNIX variants as well as on Windows. Shell seemed like a good choice when we started to write those scripts: it is available on all hosts (using Cygwin …
-
Battery life
Normally I'm quite happy with the 3h of battery life of my laptop, it covers all my disconnected time on trains etc. But it just doesn't cut it on a 10h flight, especially painful if my brain has loads of ideas to try out and things to do.
-
New Delny release
-
Resuming an scp file transfer
Sometimes you have to transfer data over dodgy links. And when you're transferring a large file this can hurt when it fails. So as explained elsewhere rsync can safe the day:
$ scp host:remote_file local_file # now interrupt and resume with $ rsync --partial --progress --rsh=ssh host:remote_file local_file
You need …
-
Should bare except statements be allowed in the Python stdlib?
Firstly to clarify the terminology, this is bare except statement:
try: ... except: ...
And this is a non-bare except statement, but bear in mind the type of the exception that is caught can be anything:
try: ... except Exception: ...
The point is that both fragments are a catch-all exception handler, only the …
-
Pain
I've spent the last 2 days trying to get a stack trace from a crashing python extension module in windows. And I still haven't figured it out. That's sooo very motivating.
Give me GNU/Linux any day.
-
cifs why don't you follow unix conventions?
Often it's nice to just have conventions and gentleman's agreements. But then sometimes someone doesn't and the harm is done, it's too hard to reverse.
NFS has introduced the de-facto standard of using <host>:<path> for the device of a remote mount. Yes this break because you can have ":" in …
-
Letters on keyboard stop working
Dear Lazy Web
I have some Very weird behaviour where the lowercase letters "C" and "V" stop generating the Correct key press event after a short while (which is why I am typing them as upper Case here...). But only when using Compiz as window manager, when switching away from …
-
PyCon UK 2009
So there's a PyCon UK 2009 Unconference. I wish I could go but have already planned to be on Pabbay and Mingulay in the Outer Hebrides, Scotland. So instead of hearing all about python I'll be clinging on to their amazing cliffs. Guess I'm not that bad off anyway...
-
How to bring a running python program under debugger control
Of course pdb has already got functions to start a debugger in the middle of your program, most notably pdb.set_trace(). This however requires you to know where you want to start debugging, it also means you can't leave it in for production code.
But I've always been envious of …
-
devenv.com via cygwin ssh (visual studio 2003)
Integrating an automatic build on a windows host with the rest of your one-command cross-platform build system can be quite a pain. Luckily there is cygwin which makes things like ssh, bash, make etc all work on windows. It's great.
The trick to building using visual studio from there is …
-
Importing modules in C extension modules
It seems that if you need another module in a function of your extension module, the way modules in the standard library seem to solve this is like this:
static PyObject * func(void) { PyObject *foo; foo = PyImport_ImportModuleNoBlock("foo"); if (foo == NULL) return NULL; /* do stuff with foo */ Py_DECREF(foo); return …
-
Singletons in a Python extension module
If you want a singleton in a C extension module for CPython you basically have to do the same as when doing this in plain Python: the .__new__() method needs to return the same object each time it is called. Implementing this has a few catches though.
static PyObject * MyType_new …
-
Voting and identification in the UK
Last Thursday I had the pleasure of being able to vote for the local council as well as for the European Parliament. Since I'm a Belgian citizen living in the United Kingdom this involved convincing the officials at the voting office to re-read their instructions (at first they only allowed …
-
Raising exceptions in threads
It's not simply raising an exception in a thread that I want. I want to raise an exception in a thread from another thread. It's like sending signals to threads, only signals in pyhon can only be delivered to the main thread (for portability). But Python has a other asynchronous …
-
New home for PSI
PSI, the Python System Information package that pulls interesting information from the kernel and makes it available in a nice Python API, has a new home at bitbucket. This means that the source conde now lives inside a mercurial repository instead of subversion.
This actually happened about a week ago …
-
Python System Information 0.3b1^W0.3b1.1
Short summary: PSI is alive and we've just released the first beta of a much improved upcoming 0.3 release!
Back in 2007 Chris Miles announced PSI - Python System Information, a Python extension module to provide access to some system information not normally available to Python. Most notably it allows …
-
Compiling applications in setup.py
If you need to compile a random application in setup.py this is not that hard:
cc = distutils.ccompiler.new_compiler() distutils.sysconfig.customize_compiler(cc) cc.link_executable(['test.c'], 'a.out')
There is no need to create the object files explicitly with cc.compile() first if you have no need for …
-
What's new in Python 2.6: logging
The What's New in Python 2.6 document is very good and contains loads of information, congrats for making it. But it misses out a nice addition to the logging module: LoggerAdapter Objects. It allows you to create a logger object with extra contextual information really easy:
log = logging.LoggerAdapter …
-
Sun Fire T1000 not closet friendly
Obviously the Sun Fire T1000 is a noisy machine, that's why it's sitting in the closet in the first place. But it has more shortcomings that make it a closet-unfriendly server:
- It shuts down at 50 degrees Celsius. That means wrapping it in a duvet to damp the noise is …
-
Time, floating points and python modules handling time
Some memory in the back of my mind tells I've once read a rant about storing time that argued you should never store time in floating point variables. My memory seems to think the rant convinced me and it does indeed seem better to store time in integers so that …
-
Reading a 64-bit core from a 32-bit proc on solaris?
I'm trying to read some pointers from a 64-bit address space file (from /proc/pid/as) while running in a 32-bit process - all this on Solaris. To do this I'm using the transitional large file compilation environment, i.e. where you get both open()/pread() and open64()/pread64() since I …
-
Replacing the screen of your digital camera
The screen of my Canon Digital Ixus 40 compact camera broke on my last trip (sport climbing in Geyikbayiri near Antalya, Turkey - if you climb: it's an amzing place!) which made me rather sad. But with the help of Ebay I found a new screen and have just managed to …
-
How to send emails
Dear World
Subject lines in emails are there for a reason.
Please use them.
Love
Floris
-
Virtual memory size (VSZ) on AIX
If you have ever looked at the ps(1) output on an AIX box and wondered why the virtual memory size (VSZ) is so small, often even smaller then the resident set size (RSS), then here's the answer: it's not the virtual memory size. It is actually the size of …
-
Closed source frustration
Developing against a closed source libraries is jut painful and wastes your time. Function definitions are missing from the header files, so you get to declare them yourself from the manpages. Then you get random error messages even though you do everything just like the manpage tells you. Some dark …
-
ssh magic
Dear Lazy Web
If I write the following in my ~/.ssh/config:
Host bartunnel HostKeyAlias bar HostName bar LocalForward 8822 foo:22 Host barjump HostKeyAlias bar HostName localhost Port 8822
Then I can connect to host bar via host foo (circumnavigating a firewall that stops me from going to bar …
-
Planning upgrades
So Debian has finally released again. Congratulations. Time to start planning upgrades of servers then.
-
Compiling 32-bit Python on amd64
If you ever feel the need to compile a 32-bit version of Python on a amd64 bit machine, this is how you do it on a Debian/Ubuntu system.
Firstly you need the correct compiler stuff, this means you need gcc-multilib and 32-bit development libraries of at least libc. On …
-
sed for binary files: bbe
While GNU sed takes care not to munge NULL characters this is not the POSIX standard, therefore it's no surprise some implementations don't manage to edit binary files properly (not to mention that many implementations will struggle with long lines). Hence the search for a binary sed:
bbe is just …
-
Specifying the RUNPATH in sofiles
When you create a shared object that depends on another shared object there are multiple ways to tell it where to look for the shared object. Two of these are by encoding (at linking time) a DT_RPATH and DT_RUNPATH entry in the .dynamic section. A third one is to use …
-
Resistance to change
Why can C developers be happy with using config.h to get constants about where to look for configuration or data files for example, yet Python developers seem to refuse to think of any way they might want to support finding these files in a portable way?
-
FreeBSD on Virtualbox
Virtualbox is my desktop virtualisation technology of choice most of the time and I wanted to have a play with FreeBSD. Seems they don't get along tough and you won't get a network connection.
- Solution 1: Change the network adaptor in Virtualbox to PCnet-PCI II (instead of PCnet-PCI III). I've …
-
datetime.datetime.totimestamp()
There exists a datetime.datetime.fromtimestamp() and many a time I've wondered why there is no .totimestamp() equivalent. The answer is that not all datetime objects can be represented by a timestamp. But if you know this restriction and want it anyway, this is the magic:
time.mktime(datetime_object.timetuple …
-
Resident Set Size (RSS) from /proc/pid/stat
Most UNIX-like systems have information about processes stored in /proc/$pid/ files, so does Linux.
If you would want to get the Resident Set Size (RSS) of a process on Linux you could find this in a number of files:
- /proc/$pid/stat
- targetted for scanf(3)
- /proc/$pid/statm …
-
Porting a C extension module to py3k
This was not that hard! First look at what everyone else has to say about this, nice aggregated on the python wiki. It covers a lot.
Now for what I discovered on top of this:
When you're defining a new type self->ob_type doesn't exist anymore. This is a problem …
-
Generating source files in setup.py
I have a Python extension module written in C and one of the weirder things it needs to do is have a number of module constants with their values which are defined (e.g. #define PREFIX_FOO 0x01 etc.) in an external C header file. All the defined names in that …
-
import skynet
So who's going to make the skynet module?
-
Finding memory leaks in Python extension modules
Valgrind is amazing, even if you've never used it before it's basic usage is really simple:
$ valgrind --tool=memcheck --leak-check=full /usr/bin/python ./test.py
This is enough to point you at the places in your extension module where you allocated stuff that didn't get freed later. This is …
-
gcc, LD_RUN_PATH & Solaris
I actually started writing a long post about how linking against shared libraries works and what and how ld.so does, but that seemed to get very long. So here the short version that assumes pre-existing knowledge.
If you want an RPATH in your DSOs (dynamic shared objects: executables and …
-
Mocking away the .__init__() method
I wanted to test a method on a class that didn't really depend on a lot of other stuff off the class. Or rather what it did depend on I had already turned into Mock objects with appropriate .return_value and asserting with .called etc. Problem was that the .__init__() method …
-
Preserving modification times in subversion
This is an oft requested feature apparently. But who know if and when it will be there. Usually things get done when someone has an itch to scratch, but given that no one has done this yet -but it's been requested for ages- it seems that it must either be …
-
Generative Tests
Both nose and py.test allow you to write generative tests. These are basically generators that yield the actual test functions and their arguments. The test runners will then test all the yielded functions with their arguments.
However I'm left wondering why this is better then just iterating in the …
-
Solaris equivalent of dpkg -S
In the past I have spent quite some time trying to figure out what package did install a certain file on Solaris. What I wanted was really just the equivalent of dpkg -S /path/to/file. But those searches where not very fruitfull, even talking to more experienced Solaris admins …
-
Bluetooth on a Toshiba Tecra A9-127
I got a Toshiba Tecra A9-127, listening to the model number of PTS52E-06002LEN as written on the back, from work to replace my dying old HP Compaq nx9030. As my laptop OS of choice is Ubuntu, currently in it's Hardy release, it's not completely coincidence that the hardware is almost …
-
Ripping videos from DVD
Physical discs are a nuisance, I really just want to play what I want to watch in the room I want to watch it just streaming it over the wireless. This actually works wonderfully well. Unfortunately just copying the file structure of a DVD works and gives you DVD-quality video …
-
Discovery of the day
Now I'm debating if I should really use that in production code...
-
Time to read standards
Sometimes I like quotes out of context...
Anyway, I don't think it really is an ambiguity in practice — only in the minds of those that have too much time to read standards documents.
—Greg Ewing (on distutils-sig)
-
@x.setter syntax in Python 2.5
The new property.setter syntax in Python 2.6 and 3.0 looks very clean to me. So I couldn't wait and wanted it in Python 2.5. With the help of comp.lang.python I finally got there (thanks especially to Arnaud Delobelle):
_property = property class property(property): def …
-
(O)OXML and ISO voting processes
Many have recently complained about ISO's voting processes, mainly how they need to be revised as currently it seems it's possible to buy yourself a standard given enough lobbyist and money.
But part of this is ISO's trust in ECMA. ISO allows ECMA to submit standards for the fast-track process …
-
Shell history
hehe
flub@signy:~$ history | awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn |head 79 nosetests 74 ls 46 cd 36 ssh 28 man 22 apt-cache 19 vi 19 ack 17 svn 17 sudo flub@signy:~$
It must be noted that most of my editing …
-
nose and ipython
I wish there was a simple way to integrate nose and ipython. Something like a command line option not to catch exceptions would be sufficient I think, so that you could do:
In [1]: %run tests/test_module.py --do-not-catch-exceptions
Obviously you'd want a shorter option switch...
Seems like Test.run …
-
One s3fs to rule them all?
Sometimes I wish we could fast forward 6 or 12 months or so. Hopefully by then there will be just one s3fs that is mature and maintained. Right now it's hard to tell which one will turn out the best.
Interstingly, of the ones that look like they could have …
-
GPL and python modules
I should probably google this, but can't think of good keywords. So instead this is a "dear lazy web" post.
Is importing a python module considered linking for the GPL? I.e. are you allowed to import a python module into an script or module that has a non-GPL-comptible license …
-
Cheating the KAME turtule
This is shocking. Even though I have not yet used IPv6 I can see the the KAME turtle dance! I won't give you a direct link but SixXS privde an IPv6->IPv4 service as well as an IPv4->IPv6 service. Feels bad having used the 4->6 bit... </p>
-
OmniORB 4.1.1 in lenny!
Something that happened about a week ago I think. But omniORB 4.1.1 finally make it into Debian testing aka lenny! This means that python-omniorb 3.1, which has been waiting for a while now, also make it into lenny!
It has been stalled for ages since we had …
-
Compiling libtool examples
This is more a reminder to myself, but here's how to compile the libtool examples:
$ cd demo $ aclocal $ libtoolize --automake $ automake --add-missing $ autoconf $ ./configure $ make
It's really how to bootstrap any GNU autotools using application I suppose.
-
My favourite package description
This sentence in a package description is always great...
You probably don't need it if you don't understand what this is all about.
Brought to you today by nut-snmp but can be found in many other packages.
-
Another shell trick
Let's start with the solution instead of the problem:
myvar=$(myfunc) || exit $?
Assuming myfunc is a function defined elsewhere in your shell script this will execute that function and assign it's output to myvar. However when myfunc fails, the entire assignment statement fails with the exit status of myfunc, so …
-
Creating a Debian ARM build host
If you want/need to compile a program on ARM but are not lucky enough to have the hardware available QEMU can still help you. Aurélien Jarno has an excellent description of how to do this. It is still valid today, even if you want to install sid instead of …
-
Docstring folding
In my last post I had a little rant about how python docstrings visually separate the function/method/class definition from the body. Mikko Ohtamaa raised a very good point though: the editor should fold it away.
Since my editor of choice is Emacs, said to be more an operating …
-
Documenting functions/methods/classes
There are several simple guidelines for writing functions or methods (guidelines that I like that is).
- Keep them clean, don't intersperse them with comments. If they require more then one or two inline comments it's too complicated and you should restructure.
- Keep them short, it should fit on one screen …
-
Paradox
Creating a Facebook group to complain about the UK ID cards because:
[...] An ID card would hold ALL personal information about you including biometrics, hospital records etc etc., a perfect target for ID thieves. [...]
And you're doing this on Facebook?
-
Mild improvement for movie industry
Until now every time I bought a DVD I found a few scary and angry leaflets in the box telling me how insanely bad I might be for buying pirated DVDs. They usually convince you that you are the worst person in the world for no particular reason. So just …
-
Rock climbing in Costa Daurada
Just spent a long week sport climbing in Costa Daurada. We stayed in a house (with about 32 students and ex-students) near the village of Cornudella de Montsant which was an excellent location really close to Siurana and Arboli. Absolutely brilliant climbing and although my brain wanted to stay a …
-
Updated omniORB in Debian
A while ago Thomas Girard started the Debian CORBA Team with the aim of maintaining omniORB and related packages in Debian. After a lot of work (and time!) we managed to get the packages of up to date versions in a good state fixing almost all bugs in Debian's BTS …
-
Seeing your browser's headers using python
-
Schneier: hammer, nail
As Bruce Schneider says:
If you remember, this was sold to the public as essential for fighting terrorism. It's already being misused.
-
Making Debian packages on an SMP machine
When you have an SMP machine (including dual core CPUs - but I'm sure everyone knows that by now), you quickly learn to use the -jN flag to GNU make(1). N is the number of CPUs you have and it lets make(1) run that many jobs in parallel whenever …
-
How far does encryption get you?
Finally, since a short while, all my hard disks are setup to use encrypted disks using LUKS. Works like a charm and you don't really notice it slow down. Unfortunately I am now left wondering what the point is, given that I currently live in the UK. Not that there …
-
Simple music players
A while ago I bragged about my own music player. Basically I couldn't find the player I liked so started creating my own. It was the first time I wrote anything related to media or a GUI -good thing my basic requirement was simplicity- and was interesting to do, but …
-
There's a first time for everying
flub@signy:~/src$ tar -xzf storm-0.10.tar.gz gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error exit delayed from previous errors flub@signy:~/src$ file storm-0.10.tar.gz storm-0.10.tar.gz: POSIX tar archive (GNU) flub@signy:~/src$ mv storm-0.10.tar …
-
EU in censor madness
So the EU seems to think that the Internet should be filtered. The first step is blocking searches for bomb recipes.
To any rational person it is obvious that this is wrong on more levels then I care to explain (others have done so already better then I can), but …
-
Capable Impressive System Crippled OS
That's my new acronym for cisco. Seriously, I did expect a learning curve for IOS when I was ordering a cisco 877W, but this is just disappointing.
No doubt IOS was once very amazing and had a very nice and clear concept and interface to the first few products it …
-
Source code to my music player
Recently I said I created an increadably stupid, dump and boring GTK music player. I also offered to make the source code available in the very unlikely case someone else would find this thing interesting. I should have known to just do that right away, so here it is.
-
My very own music player
-
Encrypted hard disk: LVM on LUKS
There are of course about a million other guides on how to use encrypted disks out there. However I did run into some trouble when trying this, so here mine. Specificly I address the issue of getting an encrypted root disk, with the root being on a Logical Volume Management …
-
Writing entropy
Q:
How long does it take to write a 120 GB disk full with random data?
A:
15 hours, 2 minutes and 27 seconds.
Obviously this depends on the machine. For me it was going at just under 8 minutes per GB, others report around 5 minutes per GB. Also …
-
Debian is 14
Happy brithday!
And thanks for everyone involved all those years.
-
Gazpacho and libglade
If you read my last post about using Gazpacho you should read the comment by Johan Dahlin too. He's one of the authors of Gazpacho and explains the libglade and save format issues^Wthings in Gazpacho nicely.
-
Making GUI applications with Gazpacho
-
Optimising python code by AST fiddling
Obviously no real optimisation, just optimisation exactly like python -O [-O] would create. But in "normal" byte compiled files, i.e. in .pyc files instead of .pyo files.
Why would I want to do that? Well, we really don't feel like shipping code with assert statements or if __debug__: ... bits …
-
How to choose a web framework?
So I have a bit of organically growing information that is nice to represent on internal web pages in a semi-organised fashion. The first iteration just runs a few python scripts from some cron jobs that utilise SimpleTAL to create staticly served pages. SimpleTAL as templating language simply because we …
-
__slots__: worth the trouble?
-
IPC in the stdlib(?)
Many interesting comments on my last musings about IPC. Each and every one of them look very nice and have their application area, go and have a look! These found a weak spot in my heart though:
- Candygram is Erlang style message passing. Very stunning, but to my liking a …
-
IPC and the GIL
As recently described excelently, threads are ugly beasts, waiting to get you when you let your guard down (and eventually you will). Yes, that means that I should really get my head round the asynchat module and stop using ThreadingMixIn with the SocketServer, but that's not the point of this …
-
What's in a number?
Ok, I coudn't resist this. Someone told me (as much as writing a blog counts as telling) this is my lucky number: 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0. Someone else says it's magic. Lucky and/or magic numbers... hmm, sounds …
-
More roundup love
As I've said before, roundup is an amazing issue tracker. Last time I wrote I had just implemented our internal bug tracker with all the weird requirements of a few developers. By now the incredibly flexible system, thanks to python, has been moulded into our complete project management tracker.
This …
-
Campaigning for -delete
Whenever find(1) is used to delete some files there are always a few variants that can be seen. They are generally combinations of -exec and -print piped to xargs. While demonstrating the increadible power of UNIX shells and pipes etc they don't appear very practical.
Most implementations struggle with …
-
Debian Developers working together
My quote of the day (from Josselin Mouette):
BTW, who said flamewars are a nuisance to the Debian project? I've never seen developers as united as when Mr Schilling is around. With a few more people like him we would probably be always working together.
-
Backup script in 43 lines
This is no more then my personal backup prefference at this moment. It is full of personal preferences and might not be what other people like. It also isn't written with portability in mind. But I think it is rather elegant.
My use case: on Friday evening I get home …
-
Timers for profilers
After some oh... and ah... experiences I have decided that the only timer that will give you correct results in a portable manner (accross POSIX platforms that is) is resource.getrusage(resource.RUSAGE_SELF) and os.times(). This is rather shocking, certainly after I then went on to discover that on …
-
Schizophrenic thoughts
On one hand I use tab-completion on the command line all the time. I even go as far as creating my own -terribly incomplete- completion functions in bash for commands that I use often
On the other hand, when programming I don't use tab-completion -maybe mostly because it's not so …
-
Roundup praise
Roundup is just an amazing bug tracking system. It really is way more general then that, their own words for it are not bad: an issue tracking system for knowledge workers.
Why I love it:
- It's written in python
- It has a poweful and flexible concept/abstraction of a database …
-
Writing applications as modules
The Problem
I recently had to write a few command line applications of the form command [options] args that did some stuff, maybe printed a few things on screen and exited with a certain exit code. Nothing weird here.
These apps where part of a larger server system however and …
-
When web pages check web-broswer compatibility
I don't trust banks and definately not online banking since their log in procedures are laugable, at least in the UK (no complaints about my Belgian bank, although not all Belgian banks are good either). But somehow I managed to get an account which only allows transactions to my own …
-
Warnings revisited (aka logging vs warnings)
Earlier I posted about how to make warning messages from warnings.warn() look like fairly normal messages so they don't give you all the details about which line of code called them. The result was neat, but I found it a fairly crude way of doing it.
There is another …
-
Sorry for the flooding...
Apologies for the flooding of planet python.
-
Overriding formatting of warnings.warn()
Today I needed to add the printing of warning messages to a command line tool written in Python. So Instead of writing things out using print, wich is not flexible enough and goes against my way to avoid user interaction deep inside functions and classes, I decided to use the …
-
Ubuntu & reportbug
-
mawk considered harmful
Consider the following little awk script:
/^--- [-[:alnum:].\/]+$/ { if ($2 !~ /[-[:alnum:].]+\/debian\//) print $2 }
According the the O'Reilly's sed & awk book it is, as far as my knowledge goes, normal valid awk. It's intended to be used like such:
~$ zcat debian_package_version.diff.gz | awk -f script.awk
The purpose is to …
-
site, eggs, etc.
Before I complained about why .pth files are not processed in files on in your $PYTHONPATH. I got a few pointers in that regard, thanks for them. And now I know why it doesn't happen: you can import foo, bar; foo.do(); bar.harm() inside a .pth file. Since this …
-
Ubuntu & upstart
Upstart is a new sysvinit </tt> replacement that Ubuntu wants to use. Scott James Remnant explained before it is innovating in that it is events based. Now he writes more about it, and one quote is this:
So what are events and where do they come from? (Note that this …
-
Nuking your home directory
/path/to/some/weird/directory$ rm -rf ~
Yep, I've done that today. It sucks. 1.5 seconds after I did that I hit the power button in the hope the kernel wouldn't have flushed it's buffers yet (hm, that old trick might not work with journaling filesystems). Boot up in …
-
When essential tools break...
I hate it when that happens. Current case:
flub@coronation:/tmp/sandbox$ gpg --clearsign test gpg: selecting openpgp failed: unknown command gpg: signing failed: general error gpg: test: clearsign failed: general error flub@coronation:/tmp/sandbox$
That's just to get desparate. Some quick poking around results in leading me to …
-
UTF-8 and Emacs
While looking around in Manoj's Debian web-page/directory for his new Python policy (for how to package Python stuff for Debian) I stumbled, amongst other intresting things, across this cool UTF-8 demo file.
To my surprise my web browser (Epiphany) shows nearly everything perfect, so does gedit. However my much …
-
site module musings
The site package reads .pth files and inserts directories listed there in sys.path. However it only scans lib directories ( lib/python2.4/site-packages/ and lib/python/site-python/ on UNIX) starting with sys.prefix or sys.exec_prefix for these .pth files. Thus if you install a module distribution using the …
-
Python web frameworks
It appears discussing web frameworks is popular currently. It also appears GvR has decided he likes Django and hopes it will become sort of the default.
I've been looking at this Django and TurboGears things for a while unable to decide if I should move the website that I appear …
-
setup.py: setup(..., extra_path="foobar", ...)
The extra_path keyword that you can pass to setup() when using the distutils for building and installing a package is completely undocumented as far as I know. I have no idea how I found it the first time (about a year ago) but now it was there and was not …
-
Today's WTF: sys.path in python != sys.path in ipython
Spot the current directory ( <snip>/build/lib.linux-i686-2.4) in the path:
<snip>/build/lib.linux-i686-2.4$ echo $PYTHONPATH /home/flub/lib/python:/home/flub/lib/python2.4: <snip>/build/lib.linux-i686-2.4$ python Python 2.4.3 (#2, Apr 27 2006, 14:43:58) [GCC 4.0.3 (Ubuntu …
-
GnuPG Smartcard on Debian Sarge
-
Dalai Lama visits Belgium
Like I care.
But China cares, that's more worrying. Dispite it not being an "official" visit (he was invited by some buddhists to open a new temple) he visited the prime minister and some other politicians. And hence the chinese ambassador says it will hurt the Belgian-China relationships as they …
-
Summer of Code 2006
There must be about a million blog posts with this title by now. Anyway, here my own little addition, I've been meaning to write this for a while.
I have not applied for SoC again althoug I was still eligible. There where intresting projects however, and in fact I'm sorry …
-
No feedback?
In the beginning of this week I emailed my mentor to know what he thought of my project, if he was happy with the state etc. I do know from his blog that he's fairly busy these days, but still would enjoy some feedback. Also just in case my email …
-
One FIXME down...
Profile.calibrate() now uses the correct timer instead of just the default timer (time.time()). In fact it still uses this timer when the user did not specify a specific timer (and why really would anyone want this? Appart form making me work one or two weeks more in the …
-
Magic
As promised I worked on a proper installble package of hprof. That went quite smooth, I had some trouble figuring out how to create a .pth file with distutils, but after some googling I found some setup keyword that worked. Not that is it documented in the official documentation. The …
-
Small update
This weeked I've been helping at home again building the patio outside. It's quite impressive to see things changing from a big pit of dirt in the morning to a nice surfaced area in the evening. If I had a digital camera I'd show you nice before and after pictures …
-
(Almost) Done!
Just completed hpstats module, it's all in CVS. This means that theoretically the project is finished! Altho I expect to code another few days on it before the end of August, nothing can be perfect... leave alone the first time round!
Along the list of TODO's is making a little …
-
Sleep is good
Last night I got some problems with a function call. As far as I could see it was right but it's unit test kept failing. After a while I called it a day and went to sleep. Wake up this morning and have a look. Only to discover it was …
-
Nice progress
Work has been progressing nicely the last few days. All the hacking has been going smoothly, which does not mean bug free - far from it! But I seem to have finally reached a good cycle of writing unit tests, implementing the features and refining that all. The scaffolding is in …
-
Ambiguous documentation
pstats does not behave as the documentation says it does. I got to the stage that you can look through the output of Lib/test/test_profile.py (in Python distribution) and the output of that very same file but with a import hprofile as profile at the top instead of …
-
Clarifying the hstats module
Some confusion seems to exist about the stats modules I'm writing. About a month and a bit ago I wrote hstats as a module to analyse hotshot profiling data. It was never my intention to make it compatible with pstats at all. It aimed at being usable to read profiling …
-
Why one should have a separate /boot partition: lessons learned
Background
The box (SPARC box to make matters more intresting) has two disks /dev/hda and /dev/hdb, they each have the same partion table appart from some free space at the end since two disks are never of the same size. All partitions are of type "RAID autodetec" and …
-
Cross platform developing
Even when using Python this is easier said then done. One should be very carefull as details lies in tiny details. Wich I discovered to my shame.
Normally I work on a desktop, this box listenst to the name of Ultra 10, so it is a UltraSparc architecture as developed …
-
Cleaning up?
So last time I said I wasn't happy with hstats. Maybe that was a bit impulsive, but still valid. Kent Johnson kindly pointed out <http://bruynooghe.blogspot.com/2005/08/i-wrote-crap.html#112378135725654786> for me that was the wrong attitude. Maybe he's right (read: yeah, of course he's right), but …
-
I wrote crap
hstats is crap. And I only wrote it a month ago...
The good part of that observation is that it shows that I'm learning. ;-)
So why really? Basically just it's desing. It has one single monolithic class. And within this class everything just happens like one would wirte precedural or …
-
Never ending quest
-
hotshot accepts user timer function
After a long day it all works. It's checked in to CVS in the "hotshot_timer-branch". The way it works is that the object returned by _hotshot.profiler(filename) now has an additional method settimer(timer). The timer is any callable object that returns a number or a sequence of numbers …
-
Pointer to funcion
Great fun these things. Certainly when the pointer to the function is stored inside a struc. The result is that my version of _hotshot now uses a void pointer in ProfilerObject as a function to get the time difference since last time it was called. For now it only goes …
-
Programming can be simple
It really is clear that it has been over 10 months since I last wrote C. As I mentioned in my last post I got segmentation faults when I changed the version string in _hotshot.c. Today I started by figuring out that the version string gets indeed used at …
-
Double Ouch
One
To get a user supplied timer function in hotshot I need to start hacking on _hotshot.c. Being cautious I pull the file from python CVS, put it in my package tree, write a setup script, build the stuff before modifying everything -just to test build environment- and finally …
-
Seconds & nanoseconds
Last night I struggled a lot with the calibration. In the end I fainted and gave up, went to bed and read a crap book for way too long -books are no good for me, even if they're crap I can't stop reading them. Today, after waking up way to …
-
The easy bits
So I've written the easy methods and functions of the profile module wrapper hprofile today. Including their unittests.
Tomorrow starts the harder work, the Profile.calibrate code. Maybe I shouldn't be too optimistic about how far I'll get with it. For what it's worth, here my current ideas of how …
-
If things do their job...
Unit tests work. It's amazing...
But good!
-
To hybernate or not to hybernate...
That's the question
Old days
For years and years I always envied my father who never had to log out of his window manager at his work, he only had to lock his screen. This meant he could leave all his terminals and text editors open at night. The next …
-
Refactor Mercilessly
Ok, I admit. Maybe I'm just using this as an excuse. I haven't really impressed myself with the amount of work done the last few day. But hey! This was my holiday after all.
The result however is that I have revised the test module for the hstats module (instead …
-
Native hoshot statistics!
Finally, after some refactoring, having cursed unittests and XP but in the end blessing them anyway. The result is there: the hstats module!
It loads hotshot data 35% faster then hotshot.stats.load() as I reported earlier. It sorts the data, and prints it out nicely. Moreover the sorting and …
-
Progress...
So I tried this XP approach of first creating unittests the last couple of days. Writing the tests goes soooo slow! I was a bit surprised on how easily I could write the real functions afterwards tho. Maybe it is good. But it requires lots of tinking in advace and …
-
Stats unittests
That's what I've mostly done, been making and polishing unittests for the hstats module (or what's written of it. hstats btw, is the module that will create the statistics for hotshot, natively reading hotshot data. In the process one extra omission (not really a bug as such ;-)) got into the …
-
Project on Savannah
So I got my project accepted on savannah. After some time of not being sure where it would be hosted, on savannah on it's own or on sourcefource in the Python project under the nondist/sandbox thing. Guess I'll stay with savannah now.
It's under the project name pyprof. Not …
-
Paperwork rant
This is going to be a rant.
I faxed Google's paperwork really early after we got it (27th of June IIRC). Now we've been told that if we haven't received confirmation yet we should re-submit! The problem is that in the mean time I've moved houses in Southampton, UK where …
-
First success!
Loading a profiling file dumped by hotshot goes *35% faster* than with hotshot's old method!
Specifically on a Sun Ultra 10 with a TI UltraSparc IIi (Sabre) processor -872BogoMips if that tells anything- I loaded the profile dump of pystones in 139.8 seconds with my new code (hstats.Stats …
-
Deep or shallow frame stack?
So I'm working on parsing the file created by hotshot. The question in the title relates to how the storing of the cumulative time has to happen. The problem is that every frame has to add it's time to the cumulative time of all the anchestors.
Shallow frame stack
Here …
-
Profiling code accounted to user code!?
It appears hotshot does only take one time sample in the profiler callback. This means that all the time spent in hotshot's code is accounted to the user code!
Mental note to self: remember this when I'm trying to increase accuracy later this summer!
-
Iterative learning
Understanding Code
I's amazing how little you (well, at least me!) understand the first time you read some (complex) code. Only after a few times you start to get the big picture and see how it all fits together. It's mostly a matter of understanding the underlying principles I guess …
-
Coding started
-
Software patents rejected!
Finally, the european parliament managed to see the light and rejected the software patents directive. There is still hope.
Also last night I found the Self-certifying Filesystem. I nerver even considered using plain NFS, but this looks very promising. Maybe tonight I'll end up trying it on my LAN. That …
-
hotshot
So I spent today looking at the old profile module and hotshot. Learned a lot - but not enough yet.
I quite like the idea of Brett to try and stick with hotshot and try to improve that. The main job would then be to write a wrapper to emulate profile …
-
Project for the next two months...
So this is my blog for the SoC (Google Summer of Code), as recommended by the email I got from the Python Software Foundation.
For those who don't know what I'm doing. It is listed on Python's Summer of Code wiki under "Profile Replacement".
I haven't done very much yet …