devork

E pur si muove

Synchronous classes in Python

Monday, November 16, 2009

What I'd like to build is an object that when doing anything with it would first acquire a lock and release it when finished. It's a pattern I use fairly-regularly and I am getting bored of always manually defining a lock next to the other object and manually acquiring and releasing it. It's also error prone.

Problem is, I can't find how to do it! The __getattribute__ method is bypassed by implicit special methods (like len() invoking .__len__()). That sucks. And from the description of this by-passing there seems to be no way to get round it. For this one time where I thought I found a use for meta-classes they still don't do the trick...

Monday, November 16, 2009 | Labels: |

11 comments:

Anonymous said...

Agree with previous poster about decorating methods. You could also use properties and decorate the properties.

Floris Bruynooghe said...

Decorating every method you need would work of course. But I really wanted to make this automatic, by simply having a mixin-type class so you could say "class Obj(dict, SyncMixin):" and get all the locking automatically.

But for that to work I need __getattribute__ to also work when calling special methods via language syntax.

Anonymous said...

This works for me :

from threading import Lock
import types

class Sync(object):
def __init__(self,f):
print("init for {0.__name__}".format(f))
self._f = f

def __call__(self,obj,*args,**kwargs):
print("Calling method {0} for obj {1}".format(self._f.__name__,obj))

if not hasattr(obj,"_lock"):
print("initiat lock for obj",obj)
# TODO: do we need to check for concurrency on this lock ?
obj._lock = Lock()


with obj._lock:
print("obj {0} is locked by {0._lock}".format(obj))
ret = self._f(obj,*args,**kwargs)
print("unlocked")

return ret

def __get__(self,obj,objtype):
return types.MethodType(self,obj)


@classmethod
def fromclass(cls,subject):
# replace every methods by a wrapper
for name,meth in subject.__dict__.items():
if hasattr(meth,'__call__'):
setattr(subject,name,Sync(meth))

return subject

if __name__ == '__main__':
@Sync.fromclass
class O(object):
def __init__(self,value):
self.value = value

def __len__(self):
return 5

def toto(self):
pass

o = O(5)
o.toto()
g = O(25)
print(o.toto())
print(g.toto())
print(len(o))


cheers ;)

--
Guillaume

Floris Bruynooghe said...

I think your solution with a class decorator has the same problem as a __getattribute__ based one would:

@Sync.fromclass
class P(dict):
pass

p = P(foo='bar')
print(len(p))
print(p['foo'])

When you try this it you can see that the lock never gets aquired for one of those calls.

Guillaume said...

with that test code :

o = O(5)
print(len(o))


I got that behavior :

# init of the class
init for toto
init for __len__
init for __init__

# Calling method __initi
Calling method __init__ for obj __main__.O object at 0xa09a6ec

# init the lock
('initiat lock for obj', __main__.O object at 0xa09a6ec)

# lock, call init, unlock
obj __main__.O object at 0xa09a6ec is locked by thread.lock object at 0xb749f100
unlocked

# call __len__
Calling method __len__ for obj __main__.O object at 0xa09a6ec

# lock, unlock
obj __main__.O object at 0xa09a6ec is locked by thread.lock object at 0xb749f100
unlocked
5

python 2.6/3.1

PS: love blogger "Your HTML cannot be accepted: Tag is not allowed: THREAD.LOCK"

Floris Bruynooghe said...

It works with the O class becaue you explicity define __len__ which gets decorated. The lookup mechanism will use your custom method (which is decorated). But if you don't explicitly define your special method then the lookup mechanism uses the builtin method and you can not decorate it or change it's lookup AFAIK. That's at least what I'm finding so far.

(Apologies for using blogger, I'm lazy)

Guillaume said...

(if it's you climbing on that wall, I don't think you are lazy)

Btw, if you do not specifiy specials methods, why do you want to lock them ?

If they come from an inherited class, there is a trivial fix, you can add them one by one,

Add this in fromclass :

for name in '__len__ __getitem__'.split():
setattr(subject,name,Sync(getattr(subject,name)))

You can also add a more "smart" loop over all parents __dict__ to add all specials methods (except perhaps new and __init__ which does not need it)

Check this http://code.activestate.com/recipes/252151/

Enjoy.

New comments are not allowed.

Subscribe to: Post Comments (Atom)