E pur si muove

Finding memory leaks in extension modules

Wednesday, November 25, 2009

After reading Ned Batchelder's post on his experience hunting a memory leak (which turned out to be a reference counting error) it occurred to me that even tough I have a script to check memory usage I should also really be checking reference counts with sys.gettotalrefcount(). And indeed, after adding this to my script I found one reference count leak. I still have faith in my script as it was before really since the reference leak in question was not making me loose memory - subtle bugs eh?

But how do you check an extension module for memory leaks? This seems pretty undocumented so here my approach:

  • First you really need a debug build of python, this helps a lot since you get to use sys.gettotalrefcount() and get more predictable memory behaviour. The most complete way to build this is something like this (the MAXFREELIST stuff adapted from this):

    ./configure --with-pydebug --without-pymalloc --prefix=/opt/pydebug \
    CPPFLAGS="-DPyDict$s -DPyTuple$s -DPyUnicode$s -DPySet$s -DPyCFunction$s -DPyList$s -DPyFrame$s -DPyMethod$s"
    make install
  • Now run the test suite using valgrind, this is troublesome but a very useful thing to do. The valgrind memory checker will help you identify problems pretty quickly. It can be confused about Python however, but you only care about your extension module so you need to filter most of this. Luckily the python distribution ships with a valgrind suppression file in Misc/valgrind-python.supp that you can use, it's not perfect but helps. This is how I invoke valgrind:

    $ /opt/pydebug/bin/python build
    $ valgrind --tool=memcheck \
        --suppression=~/python-trunk/Misc/valgrind-python.supp \
        --leak-check=full /opt/pydebug/bin/python -E -tt test
    ==8599== Memcheck, a memory error detector
    ==8599== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
    ==8599== Using Valgrind-3.5.0-Debian and LibVEX; rerun with -h for copyright info
    ==8599== Command: /opt/pydebug/bin/python -E -tt test
    ==8599== Conditional jump or move depends on uninitialised value(s)
    ==8599==    at 0x400A66E: _dl_relocate_object (do-rel.h:65)
    ==8599==    by 0x4012492: dl_open_worker (dl-open.c:402)
    ==8599==    by 0x400E155: _dl_catch_error (dl-error.c:178)
    ==8599==    by 0x4011D0D: _dl_open (dl-open.c:616)
    ==8599==    by 0x405AC0E: dlopen_doit (dlopen.c:67)
    ==8599==    by 0x400E155: _dl_catch_error (dl-error.c:178)
    ==8599==    by 0x405B0DB: _dlerror_run (dlerror.c:164)
    ==8599==    by 0x405AB40: dlopen@@GLIBC_2.1 (dlopen.c:88)
    ==8599==    by 0x8132727: _PyImport_GetDynLoadFunc (dynload_shlib.c:130)
    ==8599==    by 0x81199D9: _PyImport_LoadDynamicModule (importdl.c:42)
    ==8599==    by 0x81161FE: load_module (import.c:1828)
    ==8599==    by 0x8117FAF: import_submodule (import.c:2589)
    running test
    FAILED (failures=4, errors=2)
    ==8599== HEAP SUMMARY:
    ==8599==     in use at exit: 1,228,588 bytes in 13,293 blocks
    ==8599==   total heap usage: 280,726 allocs, 267,433 frees, 70,473,201 bytes allocated
    ==8599== LEAK SUMMARY:
    ==8599==    definitely lost: 0 bytes in 0 blocks
    ==8599==    indirectly lost: 0 bytes in 0 blocks
    ==8599==      possibly lost: 1,201,420 bytes in 13,014 blocks
    ==8599==    still reachable: 27,168 bytes in 279 blocks
    ==8599==         suppressed: 0 bytes in 0 blocks
    ==8599== Rerun with --leak-check=full to see details of leaked memory
    ==8599== For counts of detected and suppressed errors, rerun with: -v
    ==8599== Use --track-origins=yes to see where uninitialised values come from
    ==8599== ERROR SUMMARY: 75 errors from 5 contexts (suppressed: 19 from 6)

    Note that the output is very verbose, usually I actually start with --leak-check=summary. Firstly notice that valgrind gives a lot of warnings already before your extension module gets loaded, that's python's problems and not yours so skip over that. The stuff output after (and during) the output of the test suite is what interests you. Most importantly look at the definitely lost line if that's not zero the you have a leak. The possibly lost is just python's problem (which sadly might hide problems you created too). When you do have lost blocks valgrind will give you a stack trace to pinpoint it, but you'll have to swim trough lots of "possibly lost" stack traces of python to find it. Best is probably to grep for your source files in the output.

  • Next you should create function you want to execute in a loop, this should be exercising the code you want to tests for leaks. If you're really thourough possibly the entire test suite wrapped up in a function call would be good.

    Wrap it all up in a script that checks the memory usage and reference counts on each loop and compares the start and end values. Getting memory usage might be tricky from python (or you can use PSI of course) so depending on your situation you might prefer to do this with an script from your operating system.

    For PSI this is the script I currently use. I clearly have it easy since I can be sure PSI will be available :-). The reason I don't automate this script further (you could turn it into a unittest) is that I prefer to manually look at the output. Both memory and reference counting are funny and will most likely grow a little bit anyway. By looking at the output I can easily spot if it keeps growing or stabilises, there is only a problem if it keeps growing with every iteration (don't be afraid to run with many many iterations from time to time). When automating this you probably end up allowing some margin and might miss small leaks.

Hopefully some of this was useful for someone.

New Python System Information release!

Saturday, November 21, 2009

I've just released a new version of PSI! PSI is a cross-platform Python package providing real-time access to processes and other miscellaneous system information such as architecture, boottime and filesystems. Among the highlights of this release are:

  • Improved handling of time: We now have our own object to represent time. This may seem silly at first but it actually makes it easier to use all the normal ways of representing time easily as well as provide the highest possible accuracy.
  • Improved handling of process names and arguments: There is an entire wiki page dedicated to this, but basically this simplifies presenting sensible process names to users massively since some attributes will always have meaningful values.
  • Restructured exceptions: Whenever accessing attributes you will get a subclass of AttributeError like it should be so now you can happily use getattr().
  • New experimental methods on Process objects: You can now send signals to processes using the .kill() method and find their children by using the .children() method.
  • New experimental mount module: You can get detailed information about all mounted filesystems using this new module. It provides mount information as well as usage.

Another notable improvement is the ability to read the arguments of a 64-bit process while running inside a 32-bit python process on Solaris. It's small and almost no-one will notice it but make it so much more consistent!

Release early, release often: fail

Now for the bad news: all this means the API has changed in a backwards incompatible way.

It was already pretty obvious shortly after the last release that this would happen and was the reason I was hoping to release a new version soon. But that didn't happen. Although the last version had a "beta" version number on it it's trove classifier still claimed to be "alpha" and in end we don't promise API stability till we hit 1.0. But it's still not nice. Once we hit 0.3 we will actually try not to introduce changes to the API if possible. We intend to help this by using FutureWarning for APIs we're not sure about yet. In the mean time let's see how the Process API holds out during this release, hopefully it will prove to be good and require no more changes.


As before, thanks to Chris Miles and Erick Tryzelaar for helping out.

Synchronous classes in Python

Monday, November 16, 2009

What I'd like to build is an object that when doing anything with it would first acquire a lock and release it when finished. It's a pattern I use fairly-regularly and I am getting bored of always manually defining a lock next to the other object and manually acquiring and releasing it. It's also error prone.

Problem is, I can't find how to do it! The __getattribute__ method is bypassed by implicit special methods (like len() invoking .__len__()). That sucks. And from the description of this by-passing there seems to be no way to get round it. For this one time where I thought I found a use for meta-classes they still don't do the trick...

Python modules and the GPL: I still don't get it

Tuesday, November 03, 2009

I've never understood if you can use GPL python modules (or packages) in GPL-incompatibly licensed code. Today I re-read the GPL looking for this and am tempted to think yes, you can. The GPL says:

1. You may copy and distribute verbatim copies of the Program's source code as you receive it [...]

This is simple enough, what if you need to change it? This gets more interesting:

2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, [...] provided that you also meet all of these conditions:


b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.


These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.

As I uderstand this it means you can happily ship a GPL python module next to a GPL-incompatible module, even when the second uses API calls from the first. As long you are always offering to give the source of the GPL module, including modifications if you made any, you are fine. The two modules are individually identifiable sections and can be reasonably considered independent and separate works, or so I reckon at least.

But wait, the non-GPL module uses the API of the GPL module, is it still independent and separable then? In my humble opinion the GPL says it must be reasonably considered independtent and separate, so yes. It's feasable to take away the GPL module and replace it by another one that offers the same API, hence I would argue they are separable (feasable and easy are not synonyms!).

According to my reasoning it should also be legal to use GPL C libraries in non-GPL programs as long as you use them as shared libraries and not as a static library. But why does the Lesser General Public License (LGPL) exist then? Qoting the the LGPL preamble (emphasis mine):

When a program is linked with a library, whether statically or using a shared library, the combination of the two is legally speaking a combined work, a derivative of the original library. The ordinary General Public License therefore permits such linking only if the entire combination fits its criteria of freedom. The Lesser General Public License permits more lax criteria for linking other code with the library.

Now a shared C library is linked by the dynamic linker before the application gets started. If the library is not present the application will fail horribly, I guess this is why they consider it legally speaking a combined work. When using a python module the python interpreter will happily start and depending on how the application is written can fail gracefully or even provide partial functionality.

The GPL FAQ argues that it depends on the technical means used to run the final application whether the modules should both be GPL or not. The arguments seem to suggest that if code runs in shared address space as GPL-licensed code, then the whole must comply to the GPL. Given that translation of a work is considered a derrivative we can assume python translates the python code (interprets it) and then executes the results in the same address space, therefore all must comply to the GPL (including python itself).

I'm still hard pushed to call the non-GPL module a derrivative work of the GPL module however. And in a lenghty article (apparenlty written by real lawyers, definitely worth a read!) the authors argue that legally it depends on a lot more then the technicality to determine if a work becomes a derrivative work or not (and that's where it all revolves around: if its a derrivative and you distribute it the GPL applies). This does indeed seem a lot more logical: imagine you accept the interpretation in the GPL FAQ, all you need to do to distribute the two modules is use pyro or some other form of inter-process communication (excluding shared memory according to the GPL FAQ, this again I find hard to accept) and use the APIs of the modules over this IPC layer.

The above article describes how some courts have judged this derrivation question and gives a couple of pointers itself. Essentially it common sense, but that's hard to qantify as even the GPL FAQ fails. I should really summarise the factors in a paragraph here, but I was formally trained to be an engineer and not a lawyer so am not used to summarising lawyer articles (and frankly I'm too lazy to perform the exercise right now). Beside I'd probably skip a lot of subtle points so you're best off reading the article yourself.

I do have a conclusion however: if you have a python module which is not GPL-compatible but uses the API of another python module covered by the GPL chances are you are fine if: (i) you are not taking away market share of the GPL module. And (ii) you are not derriving or extending the creative work or copyrightable content of the module. But there's no distinct line, common sense is your friend (and enemy).

Subscribe to: Posts (Atom)