On 25 September 2012 19:08, Junkshops <junksh...@gmail.com> wrote: > > Can you give an example of how these data structures look after reading > only the first 5 lines? > > Sure, here you go: > > In [38]: mpef._ustore._store > Out[38]: defaultdict(<type 'dict'>, {'Measurement': > {'8991c2dc67a49b909918477ee4efd767': > <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0fe90>, > '7b38b429230f00fe4731e60419e92346': > <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0fad0>, > 'b53531471b261c44d52f651add647544': > <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0f4d0>, > '44ea6d949f7c8c8ac3bb4c0bf4943f82': > <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0f910>, > '0de96f928dc471b297f8a305e71ae3e1': > <micropheno.exchangeformat.Exceptions.FileContext object at 0x2f0f550>}}) >
Have these exceptions been raised from somewhere before being stored? I wonder if you're inadvertently keeping execution frames alive. There are some problems in CPython with this that are related to storing exceptions. > > In [39]: > mpef._ustore._store['Measurement']['b53531471b261c44d52f651add647544'].typeStr > Out[39]: 'Measurement' > Seeing how long these hex strings are, I'm confident that you would save a significant amount of memory by converting them to int. > > In [40]: > mpef._ustore._store['Measurement']['b53531471b261c44d52f651add647544'].lineNumber > Out[40]: 5 > > In [41]: mpef._ustore._idstore > Out[41]: defaultdict(<class 'micropheno.exchangeformat.KBaseID.IDStore'>, > {'Measurement': <micropheno.exchangeformat.KBaseID.IDStore object at > 0x2f0f950>}) > > In [43]: mpef._ustore._idstore['Measurement']._SIDstore > Out[43]: defaultdict(<function <lambda> at 0x2ece7d0>, {'emailRemoved': > defaultdict(<function <lambda> at 0x2c4caa0>, {'microPhenoShew2011': > defaultdict(<type 'dict'>, {0: {'MLR_124572462': > '8991c2dc67a49b909918477ee4efd767', 'MLR_124572161': > '7b38b429230f00fe4731e60419e92346', 'SMMLR_12551352': > 'b53531471b261c44d52f651add647544', 'SMMLR_12551051': > '0de96f928dc471b297f8a305e71ae3e1', 'SMMLR_12550750': > '44ea6d949f7c8c8ac3bb4c0bf4943f82'}})})}) > Also I think lambda functions might be able to keep the frame alive. Are they by any chance being created in a function that is called in a loop? >>> def f(): ... x = 4 ... return lambda : x ... >>> g = f() >>> g() # Accesses local variable from kept-alive frame 4 >>> x Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'x' is not defined Oscar
-- http://mail.python.org/mailman/listinfo/python-list