Re: Automatic reloading, metaclasses, and pickle
On Feb 27, 6:47 pm, Ziga Seilnacht [EMAIL PROTECTED] wrote: Andrew Felch wrote: Thanks for checking. I think I narrowed the problem down to inheritance. I inherit from list or some other container first: class PointList( list, AutoReloader ): def PrintHi1(self): print Hi2 class MyPrintingClass( AutoReloader ): def PrintHi2(self): print Hi2v2 Automatic reloading works for MyPrintingClass but not for PointList. Any ideas? -Andrew Ah yes, this is the problem of list.__new__ not calling the next class in MRO. Try to switch the bases, so that AutoReloader's __new__ method will be called first. Ziga- Hide quoted text - - Show quoted text - So there is another bug crippling Python (see https://sourceforge.net/tracker/?func=detailatid=105470aid=628925group_id=5470 ). Pickle and cPickle get confused after reloading, and won't let you dump the objects to file using protocol 2. The bug occurred for me when an instance of an AutoReloader class had a member not derived from AutoReloader (error text: it's not the same object ). The workaround is to use text mode, which I assume means not using protocol 2. After that, you can reload them in text mode, and then dump them using protocol 2. This is not so bad since you only do it when you were going to dump objects to file anyway, so it really only takes 3x as long on those occassions. The metaclass adaptation is still incredibly useful because it avoids having to do pickle dump- loads when you wouldn't otherwise have had to (approaching an infinite speedup in those cases, thanks Ziga! :-). Another caveat is that pickle dumping doesn't work if there is a pointer to a global function (that changes) that is a member of an instance of a class derived from AutoLoader. The error message has subtext: it's not the same object . That doesn't seem so bad to me, since you don't have to do that type of thing (function pointers) so much now that you can use classes! The best solution would be to make pickle protocol 2 have the same behavior as the text protocol (i.e. actually work in this case). Regarding the function pointers, anybody (Ziga :-) know why it is that pickle chokes so hard on this? Is there some adaptation of the metaclass or a less beautiful hack that could fix it? Thanks to those readers following me down the rabbit hole. Ziga's metaclass adaptation has saved Python in my mind, and these nits seem to be worth discussing. - Andrew Felch -- http://mail.python.org/mailman/listinfo/python-list
Automatic reloading, metaclasses, and pickle
Hello all, I'm using the metaclass trick for automatic reloading of class member functions, found at: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164 My problem is that if I 1) pickle an object that inherits from AutoReloader 2) unpickle the object 3) modify one of the pickled' object's derived class methods 4) reload the module holding the class ... then the changes don't affect the unpickled object. If I unpickle the object again, of course the changes take effect. My friend that loves Smalltalk is laughing at me. I thought I had the upperhand when I discovered the metaclasses but now I am not sure what to do. I really don't want to have to unpickle again, I'm processing video and it can take a long time. By the way, I used to avoid all of these problems by never making classes, and always building complex structures of lists, dictionaries, and tuples with global functions. It's going to take me a while to kick those horrible habits (during my transition, I'm deriving from list, dict, etc. hehe), perhaps a link to the metaclass trick is in order in the tutorial's comments on reload? Any help that avoids having to unpickle again is appreciated! Thanks, Andrew Felch -- http://mail.python.org/mailman/listinfo/python-list
Re: Automatic reloading, metaclasses, and pickle
On Feb 27, 3:50 pm, Ziga Seilnacht [EMAIL PROTECTED] wrote: Andrew Felch wrote: Hello all, I'm using the metaclass trick for automatic reloading of class member functions, found at:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164 My problem is that if I 1) pickle an object that inherits from AutoReloader 2) unpickle the object 3) modify one of the pickled' object's derived class methods 4) reload the module holding the class ... then the changes don't affect the unpickled object. If I unpickle the object again, of course the changes take effect. My friend that loves Smalltalk is laughing at me. I thought I had the upperhand when I discovered the metaclasses but now I am not sure what to do. I really don't want to have to unpickle again, I'm processing video and it can take a long time. By the way, I used to avoid all of these problems by never making classes, and always building complex structures of lists, dictionaries, and tuples with global functions. It's going to take me a while to kick those horrible habits (during my transition, I'm deriving from list, dict, etc. hehe), perhaps a link to the metaclass trick is in order in the tutorial's comments on reload? Any help that avoids having to unpickle again is appreciated! Thanks, Andrew Felch This happens because unpickling doesn't recreate your object by calling its type. MetaInstanceTracker registers an instance only when it is created by calling a class. You can solve this by moving the instance registration to AutoReloader.__new__ and using pickle protocol version 2, but the best solution is to avoid both pickle (old pickles break if you change your code) and autoreloading (it's meant to be used in interactive console and entertaining ircbots, not in normal code). Ziga- Hide quoted text - - Show quoted text - Thanks Ziga. I use pickle protocol 2 and binary file types with the command: cPickle.dump(obj, file, 2) I did your suggestion, i commented out the __call__ function of MetaInstanceTracker and copied the text to the __new__ function of AutoReloader (code appended). I got a crazy recursive error message (also appended below). In my code, I am creating a new instance, rather than using the pickled object (it needs to work in both modes). Thanks very much for helping me get through this. With my development approach, finding a solution to this problem is really important to me. ... File C:\Python25\lib\site-packages\tdbu.py, line 67, in __new__ instance = super(MetaInstanceTracker, self).__call__(*args, **kw) File C:\Python25\lib\site-packages\tdbu.py, line 67, in __new__ instance = super(MetaInstanceTracker, self).__call__(*args, **kw) File C:\Python25\lib\site-packages\tdbu.py, line 67, in __new__ instance = super(MetaInstanceTracker, self).__call__(*args, **kw) File C:\Python25\lib\site-packages\tdbu.py, line 67, in __new__ instance = super(MetaInstanceTracker, self).__call__(*args, **kw) File C:\Python25\lib\site-packages\tdbu.py, line 67, in __new__ instance = super(MetaInstanceTracker, self).__call__(*args, **kw) File C:\Python25\lib\site-packages\tdbu.py, line 67, in __new__ instance = super(MetaInstanceTracker, self).__call__(*args, **kw) RuntimeError: maximum recursion depth exceeded code import weakref, inspect class MetaInstanceTracker(type): def __new__(cls, name, bases, ns): t = super(MetaInstanceTracker, cls).__new__(cls, name, bases, ns) t.__instance_refs__ = [] return t def __instances__(self): instances = [(r, r()) for r in self.__instance_refs__] instances = filter(lambda (x,y): y is not None, instances) self.__instance_refs__ = [r for (r, o) in instances] return [o for (r, o) in instances] ##def __call__(self, *args, **kw): ##instance = super(MetaInstanceTracker, self).__call__(*args, **kw) ##self.__instance_refs__.append(weakref.ref(instance)) ##return instance class InstanceTracker: __metaclass__ = MetaInstanceTracker class MetaAutoReloader(MetaInstanceTracker): def __new__(cls, name, bases, ns): new_class = super(MetaAutoReloader, cls).__new__( cls, name, bases, ns) f = inspect.currentframe().f_back for d in [f.f_locals, f.f_globals]: if d.has_key(name): old_class = d[name] for instance in old_class.__instances__(): instance.change_class(new_class) new_class.__instance_refs__.append( weakref.ref(instance)) # this section only works in 2.3 for subcls in old_class.__subclasses__(): newbases = () for base in subcls.__bases__: if base is old_class: newbases += (new_class,) else:
Re: Automatic reloading, metaclasses, and pickle
On Feb 27, 5:30 pm, Ziga Seilnacht [EMAIL PROTECTED] wrote: Andrew Felch wrote: Thanks Ziga. I use pickle protocol 2 and binary file types with the command: cPickle.dump(obj, file, 2) I did your suggestion, i commented out the __call__ function of MetaInstanceTracker and copied the text to the __new__ function of AutoReloader (code appended). I got a crazy recursive error message (also appended below). In my code, I am creating a new instance, rather than using the pickled object (it needs to work in both modes). Thanks very much for helping me get through this. With my development approach, finding a solution to this problem is really important to me. Here is a version that should work. It should work with all protocols, see InstanceTracker.__reduce_ex__. Note that all subclasses of InstanceTracker and AutoReloader should be cautious when overriding the __new__ method. They must call their base class' __new__ method, preferably by using super(), or the tracking won't work. # adapted fromhttp://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/160164 import weakref, inspect class MetaInstanceTracker(type): def __init__(cls, name, bases, ns): super(MetaInstanceTracker, cls).__init__(name, bases, ns) cls.__instance_refs__ = [] def __instances__(cls): instances = [] validrefs = [] for ref in cls.__instance_refs__: instance = ref() if instance is not None: instances.append(instance) validrefs.append(ref) cls.__instance_refs__ = validrefs return instances class InstanceTracker(object): __metaclass__ = MetaInstanceTracker def __new__(*args, **kwargs): cls = args[0] self = super(InstanceTracker, cls).__new__(*args, **kwargs) cls.__instance_refs__.append(weakref.ref(self)) return self def __reduce_ex__(self, proto): return super(InstanceTracker, self).__reduce_ex__(2) class MetaAutoReloader(MetaInstanceTracker): def __init__(cls, name, bases, ns): super(MetaAutoReloader, cls).__init__(name, bases, ns) f = inspect.currentframe().f_back for d in [f.f_locals, f.f_globals]: if name in d: old_class = d[name] for instance in old_class.__instances__(): instance.change_class(cls) cls.__instance_refs__.append(weakref.ref(instance)) for subcls in old_class.__subclasses__(): newbases = [] for base in subcls.__bases__: if base is old_class: newbases.append(cls) else: newbases.append(base) subcls.__bases__ = tuple(newbases) break class AutoReloader(InstanceTracker): __metaclass__ = MetaAutoReloader def change_class(self, new_class): self.__class__ = new_class Ziga I pasted the code into mine and replaced the old. It seems not to work for either unpickled objects or new objects. I add methods to a class that inherits from AutoReloader and reload the module, but the new methods are not callable on the old objects. Man! It seems we're so close, it will be huge if this ends up working. This stuff is so over my head, I wish I could help in some additional way. -Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Automatic reloading, metaclasses, and pickle
On Feb 27, 3:23 pm, Ziga Seilnacht [EMAIL PROTECTED] wrote: Andrew Felch wrote: I pasted the code into mine and replaced the old. It seems not to work for either unpickled objects or new objects. I add methods to a class that inherits from AutoReloader and reload the module, but the new methods are not callable on the old objects. Man! It seems we're so close, it will be huge if this ends up working. This stuff is so over my head, I wish I could help in some additional way. -Andrew Did you copy and paste the entire module? I fiddled almost all parts of the original code. I did some base testing and it worked for me. Could you post the traceback? Note that Google Groups messed up indentation; in MetaAutoReloader.__init__, the line starting with cls.__instance_refs__ should be at the same level as previous line. Did you restart Python? InstanceTracker, MetaInstanceTracker and MetaAutoReloader are not auto reloaded :). Ziga Thanks for checking. I think I narrowed the problem down to inheritance. I inherit from list or some other container first: class PointList( list, AutoReloader ): def PrintHi1(self): print Hi2 class MyPrintingClass( AutoReloader ): def PrintHi2(self): print Hi2v2 Automatic reloading works for MyPrintingClass but not for PointList. Any ideas? -Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Automatic reloading, metaclasses, and pickle
On Feb 27, 3:47 pm, Ziga Seilnacht [EMAIL PROTECTED] wrote: Andrew Felch wrote: Thanks for checking. I think I narrowed the problem down to inheritance. I inherit from list or some other container first: class PointList( list, AutoReloader ): def PrintHi1(self): print Hi2 class MyPrintingClass( AutoReloader ): def PrintHi2(self): print Hi2v2 Automatic reloading works for MyPrintingClass but not for PointList. Any ideas? -Andrew Ah yes, this is the problem of list.__new__ not calling the next class in MRO. Try to switch the bases, so that AutoReloader's __new__ method will be called first. Ziga That did it! Thanks so much! This really seems to extend Python to be a much better rapid prototyping langauge. It certainly allows me to now use classes where I avoided them like the plague before. Perhaps I'll be able to make programs 1500 lines now! :-) -Andrew -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to have a for-loop index?
Sometimes C++ is the right tool/style for the job, but I don't need the speed or efficiency of C++. -- http://mail.python.org/mailman/listinfo/python-list
Best way to have a for-loop index?
I write a lot of code that looks like this: for myElement, elementIndex in zip( elementList, range(len(elementList))): print myElement , myElement, at index: ,elementIndex My question is, is there a better, cleaner, or easier way to get at the element in a list AND the index of a loop than this? TIA, Andrew -- http://mail.python.org/mailman/listinfo/python-list
Numarray - Using Integers Efficiently for Boolean Values
Below is the code to/from Boolean arrays and Unsigned integers. On my Pentium 4, functions such as bitwise_and are 32 times faster when run on 32-bit integers instead of the entire-byte-consuming-Boolean. Good luck all:-) uint32Mask = numarray.array([0x0001,0x0002,0x0004,0x0008, \ 0x0010,0x0020,0x0040,0x0080, \ 0x0100,0x0200,0x0400,0x0800, \ 0x1000,0x2000,0x4000,0x8000, \ 0x0001,0x0002,0x0004,0x0008, \ 0x0010,0x0020,0x0040,0x0080, \ 0x0100,0x0200,0x0400,0x0800, \ 0x1000,0x2000,0x4000,0x8000], numarray.UInt32) uint32MaskInner = numarray.copy.deepcopy(uint32Mask) uint32MaskInner.shape = [32,1] uint32MaskOuter = numarray.copy.deepcopy(uint32Mask) uint32MaskOuter.shape = [1,32] def BoolToUInt32(myArr): if myArr.size()%32 != 0: print Size is: , myArr.size() return numarray.matrixmultiply(numarray.reshape(myArr,[myArr.size()/32,32]),uint32MaskInner).flat def UInt32ToBool(myArr,destination=None): if destination == None: destination = numarray.zeros([myArr.size()*32],numarray.Bool) #return numarray.bitwise_and(numarray.reshape(myArr,[myArr.size(),1]),uint32MaskOuter).flat #else: destination.shape = [myArr.size(),32] numarray.bitwise_and(numarray.reshape(myArr,[myArr.size(),1]),uint32MaskOuter,destination) destination.shape = [destination.size()] return destination Test of code: import numarray gram.UInt32ToBool(ni) array([1, 0, 0, ..., 0, 0, 0], type=Bool) numarray.all(numarray.equal(n,gram.UInt32ToBool(gram.BoolToUInt32(n 1 n array([1, 0, 0, ..., 0, 0, 0], type=Bool) n.shape (1024,) numarray.all(numarray.equal(n,gram.UInt32ToBool(gram.BoolToUInt32(n 1 -- http://mail.python.org/mailman/listinfo/python-list
Easy way to make Numarray Bool use 1 bit?
I need to be able to store thousands of one-million-length bit arrays. The first operations use the operator. But eventually I must perform an inner product with a one-million-length integer array (summing at all the indices of the integer operand where the boolean operand has a 1). If I use the Boolean type, I end up using 1-byte, per needed bit, which is not efficient enough (only 1,000 arrays per gigabyte of memory). I'm limited by 32-bit windows to use only 2 GB total. Anybody know of a fast way to use only 125 kilobytes of memory per array and yet be able to inner product with an integer array quickly? Don't tell me I have to write C/C++ ! (ugh!) TIA, AndrewF -- http://mail.python.org/mailman/listinfo/python-list
Re: Easy way to make Numarray Bool use 1 bit?
Ahh, I figured it out. The key is to set up the shapes so they broadcast like an outerproduct, but using bitwise_and as the operation and the second operand as the bit filter. So we temporarily expand each bit to a byte for numarray to be able to use it for dot products. If you use a temporary Boolean variable to store the bitwise_and ufunc output, the answer is automatically converted to boolean. Set up the shapes for an innerproduct-like operation to go from byte format back to bit compact Uint8 format. Go numarray! -- http://mail.python.org/mailman/listinfo/python-list
