E pur si muove

Virtual memory size (VSZ) on AIX

Saturday, February 28, 2009

If you have ever looked at the ps(1) output on an AIX box and wondered why the virtual memory size (VSZ) is so small, often even smaller then the resident set size (RSS), then here's the answer: it's not the virtual memory size. It is actually the size of the data section of the virtual memory of the process.

If you really do want to know the VSZ of a process you'll have to use svmon -P 123, there you will find it. But do multiply this by the pagesize (pagesize(1) is a user tool on AIX). If only somehow this would be documented.

Oh, and if someone knows where the source of svmon lives I'd love to know how they actually find that, using the struct procent64 structure from procinfo.h I can not figure out a way of finding it. And the VSZ information in /proc/123/psinfo is completely whacky, no idea what that is supposed to be.

For completeness sake: this is on AIX 5.3.

Closed source frustration

Thursday, February 26, 2009

Developing against a closed source libraries is jut painful and wastes your time. Function definitions are missing from the header files, so you get to declare them yourself from the manpages. Then you get random error messages even though you do everything just like the manpage tells you. Some dark corners of the internet then suggest one of the parameters is actually the 32-bit variant of the type instead of the generic one. Still no luck.

Just let me look inside the code to figure out why it's not working, thank you very much.

This post was brought to you by the joys of the getprocs64(3) call on AIX 5.3. Any hints on it's usage would be appreciated.

Update: It finally works!

ssh magic

Thursday, February 19, 2009

Dear Lazy Web

If I write the following in my ~/.ssh/config:

Host bartunnel
  HostKeyAlias bar
  HostName bar
  LocalForward 8822 foo:22

Host barjump
  HostKeyAlias bar
  HostName localhost
  Port 8822

Then I can connect to host bar via host foo (circumnavigating a firewall that stops me from going to bardirectly) just like am connecting to it directly. E.g. in two separate shells (in this order):

$ ssh bartunnel # this sets up the tunnel
# different shell (or use -n on the last one)
$ ssh barjump # now I'm connected normally

Now is there something I could write in my ssh configuration file that I could just do this in one step? I want to simply do:

$ ssh barjump

and the tunnel should be set up for me in the background. Likewise if I close the connection the tunnel should go. Is this possible?

Compiling 32-bit Python on amd64

Friday, February 13, 2009

If you ever feel the need to compile a 32-bit version of Python on a amd64 bit machine, this is how you do it on a Debian/Ubuntu system.

Firstly you need the correct compiler stuff, this means you need gcc-multilib and 32-bit development libraries of at least libc. On Debian/Ubuntu installing the gcc-multilib package will pull in most if not all of the required dependencies.

Next is invoking the configure script of Python. Sadly Python is one of those autoconf using projects who advertise the use of environment variables like CFLAGS in the --help output of ./configure but don't actually respect them, this is all too common for autoconf-but-no-automake using projects. So the correct way to start building is using the OPT environment variable instead of CFLAGS:

OPT=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/pym32

You may want to finetune the OPT value, since this is where normally -g -O3 etc appears in so you've just got rid of those. I'm not quite convinced of this design but anyway.

Now you can watch the compilation, depending on your machine you may have time to make a cup of tea or so. Near the end you'll see something like this:

Failed to build these modules:
_hashlib           _sqlite3           _ssl
_tkinter           bz2                gdbm

This is pretty much exactly the same as when normally compiling python: go find the packages that provide the development libraries needed for these modules. Only this time you need to look for the 32-bit variety of these. On Debian/Ubuntu look out for packages named like lib32foo-dev. After installing all applicable ones I could find this is what I got it down too in the end (only using system packages):

Failed to build these modules:
_hashlib           _sqlite3           _ssl
_tkinter           gdbm

Just in case you aren't quite happy with your achievement so far you could now try compiling an extension module against your 32-bit python:

LDFLAGS=-m32 /opt/pym32/bin/python build

Now wasn't this useful? Silly binary-only libraries...

sed for binary files: bbe

Friday, February 13, 2009

While GNU sed takes care not to munge NULL characters this is not the POSIX standard, therefore it's no surprise some implementations don't manage to edit binary files properly (not to mention that many implementations will struggle with long lines). Hence the search for a binary sed:

bbe is just that. It allows you to do operations on blocks --you can define the block size in a variety of flexible ways-- as well as on bytes. So the most simple substitution command from sed just looks exactly the same. It's gorgeous.

It is a C program that you'll need to compile, but has no build dependencies so you can use it pretty much anywhere.

Specifying the RUNPATH in sofiles

Monday, February 09, 2009

When you create a shared object that depends on another shared object there are multiple ways to tell it where to look for the shared object. Two of these are by encoding (at linking time) a DT_RPATH and DT_RUNPATH entry in the .dynamic section. A third one is to use the LD_LIBRARY_PATH environment variable (at runtime). The order of precedence of these is:

  1. RPATH

The question is how to encode these into your shared object. This is normally done with the --rpath or -R options to the linker. But as the name suggest this will create an RPATH. When using GNU ld(1) you add an --enable-new-dtags option which does add the (newer) RUNPATH, and when both RPATH and RUNPATH are present the runtime linker will ignore the RPATH. On Solaris however the linker does add a RUNPATH by default as soon as you use -R, thus you will always end up with LD_LIBRARY_PATH overriding the value you gave with -R. Good to keep that in mind.

Subscribe to: Posts (Atom)