devork

E pur si muove

Writing applications as modules

Wednesday, February 14, 2007

The Problem

I recently had to write a few command line applications of the form "command [options] args" that did some stuff, maybe printed a few things on screen and exited with a certain exit code. Nothing weird here.

These apps where part of a larger server system however and needed to use some of the modules from these servers for some of their work (in the name of code reuse obviously). A little later these apps would look nicer when they are separated out into their own modules as well (all hail code reuse again the apps can share code) and now it is really a short step to wanting to use some of the more general modules of the apps in the server.

I'm not sure that last step was very important, I think it all started when the app was split up in modules. But the last one made it very obvious: you can't just print random stuff to the user and decide to sys.exit() the thing anywhere you want. You want the code to behave like real modules: throw exceptions and not print anything on the terminal. That's not all, you also want to write unit tests for every bit of code too. Ultimately you need one main routine and you want to test that too, so even that can't exit the program.

The Solution

Executable Wrapper

The untestable code needs to remain to an absolute minimum. Code is untestable (ok, there are work arounds) when it sys.exit()s so I raise exceptions instead. I defined exceptions as such:

class Exit(Exception):
   def __init__(self, status):
       self.status = status

   def __str__(self):
       return 'Exit with status: %d' % self.status

class ExitSucess(Exit):
   def __init__(self):
       Exit.__init__(self, 0)

class ExitFailure(Exit):
   def __init__(self):
       Exit.__init__(self, 1)

This allows for a very small executable wrapper:

#!/usr/bin/env python

import sys
from mypackage.apps import myapp

try:
        myapp.main()
except myapp.Exit, e:
        sys.exit(e.status)
except Exception, e:
        sys.stderr.write('INTERNAL ERROR: ' + str(e) + '\n')
        sys.exit(1)

The last detail is having main() defined as def mypackage.myapp.main(args=sys.argv) for testability, but that's really natural.

Messages for the user

These fall broadly in two categories: (1) short warning messages and (2) printing output. The second type is easily limited to a few very simple functions that do little more then just a few print statements, help() is an obvious example. For the first there is the logging module. In our case the logging module is used almost everywhere in the server code anyway, but even if it isn't it is a convenient way to be able to silence the logging. It's default behaviour is actually rather useful for an application, all that's needed is something like:

import logging

logging.basicConfig(format='%(levelname)s: %(message)s')

The lovely thing about this that you get --verbose or --quiet almost for free.

Mixing it together

This one handles fatal problems the program detects. You could just do a logging.error(msg) followed by a raise ExitFailure. But this just doesn't look very nice, certainly not outside the main app module (mypackages.apps.myapp in this case). But a second option is to do something like

 raise MyFatalError, 'message to user'

And have inside the main() another big try...except block:

try:
        workhorse(args)
except FatalError, e:
        sys.stderr.write('ERROR: ' + str(e) + '\n')
        raise ExitFailure

Just make sure FatalError is the superclass of all your fatal exceptions and that they all have a decent __str__() method. The reason I like this is that it helps keeping fatal error messages consistent wherever you use them in the app, as all the work is done inside the __str__() methods.

One final note; when using the optparse module you can take two stances: (1) "optparse does the right thing and I don't need to debug it or write tests for it" or (2) "I'm a control freak". In the second case you can subclass the OptionParser and override it's error() and exit() methods to conform to your conventions.

Wednesday, February 14, 2007 | Labels: |

2 comments:

PJE said...

Two points:

First, you don't need FatalError. raising SystemExit("message") does the same thing already, and sys.exit(arg) raises SystemExit(arg), so you'll find that sys.exit("message") is sufficient. What's more, raising a SystemExit(arg) where arg is not an integer or None, will automatically print arg to stderr as the program exits.

Thus, if you want to trap the exit, you can use "except SystemExit,v", and v.args[0] will be the message (or None, or whatever the sys.exit() argument was).

Second, and this is just an FYI, setuptools will automatically wrap a "main" function in a script wrapper like this, and it automatically calls sys.exit() on the function's return value, so it can return an integer exit code, a string error message, or None to exit normally. So, if you like developing in this style, you might want to check that out, especially since it automatically handles making the appropriate wrapper script by platform: for example, it generates .exe wrappers on Windows, and plain (no file-extension) #! scripts everywhere else.

Floris Bruynooghe said...

The SystemExit is rather cute, I didn't realise you could do that. However I'd still want to use subclasses of it as it is nice to have diversify between the error in case the module it originates from is not used in the normal application. Also overwriting __str__() or FalalError allows more consistent error messages. Lastly I get a nice list of all possible fatal errors in one place. I agree that it's all very subjective however.

As for the second point, nice to know setuptools agrees with my way of building a script. I'll have to check that out sometime.

New comments are not allowed.

Subscribe to: Post Comments (Atom)