Re: Memory footpring of python objects
BlueBird wrote: > I have a program that manages several thousands instances of one > object. To reduce memory > consumption, I want of course that specific object to have the > smallest memory footpring possible. > > I have a few ideas that I want to experiment with, like using > __slots__, using a tuple or using a dict. My > question is: how do I know the memory footprint of a given python > object ? I could not find any > builtin functions for this in the documentation. I managed to 1/3 the memory requirement of our program by using __slots__ on the most common class (several hundred thousand instances!). When doing these optimisations I ran a repeatable script and measured the total memory usage using the OS tools (top in my case). -- Nick Craig-Wood -- http://www.craig-wood.com/nick -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory footpring of python objects
On 22 avr, 11:56, Steven D'Aprano wrote: > On Wed, 22 Apr 2009 02:30:32 -0700, Chris Rebert wrote: > > On Wed, Apr 22, 2009 at 2:24 AM, BlueBird wrote: > > >> Hi, > > >> I have a program that manages several thousands instances of one > >> object. To reduce memory > >> consumption, I want of course that specific object to have the smallest > >> memory footpring possible. > > >> I have a few ideas that I want to experiment with, like using > >> __slots__, using a tuple or using a dict. My question is: how do I know > >> the memory footprint of a given python object ? I could not find any > >> builtin functions for this in the documentation. > > > sys.getsizeof() -http://docs.python.org/library/sys.html#sys.getsizeof > > Only in Python 2.6. > > But if you search Activestate, there's a recipe to do the same thing. Ah, > here it is: > > http://code.activestate.com/recipes/546530/ Thanks. "There is always a tool that makes you realise how you are less smart than you thought" - Andrew Morton, about git. I had a bit of hard time getting through the different sizes reported and the vocabulary. From what I understood, basicsize is the size of the object alone, without the size of his references. And asizeof( x, code=False, limit=100 ) will give me the memory footprint of a new instance of an object. In my case, I just want an object with named attributes. My two attempts for comparison: import sys, asizeof def print_sizeof( v_name, v ): print 'Size of %s: %d, %d' % (v_name, asizeof.flatsize(v), asizeof.asizeof(v, code=False, limit=100) ) def test_impl( v ): assert v.a == 1 assert v.b == 'bbb' assert v.c == False ### The reference, a dictionnary ref_dict = {'a':1, 'b':'bbb', 'c':False } print_sizeof( 'ref_dict', ref_dict ) ### A modified dictionnary class AttrDict (dict): """Dictionary allowing attribute access to its elements if they are valid attribute names and not already existing methods.""" def __getattr__ (self, name): return self[name] attrDict = AttrDict( ref_dict ) test_impl( attrDict ) print_sizeof( 'attrDict', attrDict ) ### Using __slots__ class ObjWithSlots(object): __slots__ = [ 'a', 'b', 'c' ] objWithSlots = ObjWithSlots() for k in ref_dict: setattr( ObjWithSlots, k, ref_dict[k] ) test_impl( objWithSlots ) print_sizeof( 'ObjectWithSlots', objWithSlots ) is giving me: Size of ref_dict: 140, 304 Size of attrDict: 140, 304 Size of ObjectWithSlots: 36, 144 So, it looks like I could shrink down the structure to 144 bytes in a simple case. Marco, my objects have varying length data structures (strings) so I don't think that would be possible. And it would clearly be overkill in my case. -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory footpring of python objects
On Wed, 22 Apr 2009 02:30:32 -0700, Chris Rebert wrote: > On Wed, Apr 22, 2009 at 2:24 AM, BlueBird wrote: >> >> Hi, >> >> I have a program that manages several thousands instances of one >> object. To reduce memory >> consumption, I want of course that specific object to have the smallest >> memory footpring possible. >> >> I have a few ideas that I want to experiment with, like using >> __slots__, using a tuple or using a dict. My question is: how do I know >> the memory footprint of a given python object ? I could not find any >> builtin functions for this in the documentation. > > sys.getsizeof() - http://docs.python.org/library/sys.html#sys.getsizeof Only in Python 2.6. But if you search Activestate, there's a recipe to do the same thing. Ah, here it is: http://code.activestate.com/recipes/546530/ -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory footpring of python objects
BlueBird wrote: I have a program that manages several thousands instances of one object. To reduce memory consumption, I want of course that specific object to have the smallest memory footpring possible. Have you thought of using something like the flyweight pattern and a compact data representation like a numpy array? -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory footpring of python objects
On Wed, Apr 22, 2009 at 2:24 AM, BlueBird wrote: > > Hi, > > I have a program that manages several thousands instances of one > object. To reduce memory > consumption, I want of course that specific object to have the > smallest memory footpring possible. > > I have a few ideas that I want to experiment with, like using > __slots__, using a tuple or using a dict. My > question is: how do I know the memory footprint of a given python > object ? I could not find any > builtin functions for this in the documentation. sys.getsizeof() - http://docs.python.org/library/sys.html#sys.getsizeof Cheers, Chris -- I have a blog: http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list