Re: Memory usage steadily going up while pickling objects

2013-06-15 Thread dieter
Giorgos Tzampanakis writes: > ... > So it seems that the pickle module does keep some internal cache or > something like that. This is highly unlikely: the "ZODB" (Zope object database) uses pickle (actually, it is "cPickle", the "C" implementation of the "pickle" module) for serialization. The "

Re: Memory usage steadily going up while pickling objects

2013-06-15 Thread Giorgos Tzampanakis
On 2013-06-15, Peter Otten wrote: > Giorgos Tzampanakis wrote: > >> So it seems that the pickle module does keep some internal cache or >> something like that. > > I don't think there's a global cache. The Pickler/Unpickler has a per- > instance cache (the memo dict) that you can clear with the c

Re: Memory usage steadily going up while pickling objects

2013-06-15 Thread Peter Otten
Giorgos Tzampanakis wrote: > So it seems that the pickle module does keep some internal cache or > something like that. I don't think there's a global cache. The Pickler/Unpickler has a per- instance cache (the memo dict) that you can clear with the clear_memo() method, but that doesn't matter

Re: Memory usage steadily going up while pickling objects

2013-06-15 Thread Giorgos Tzampanakis
On 2013-06-15, Dave Angel wrote: > On 06/14/2013 07:04 PM, Giorgos Tzampanakis wrote: >> I have a program that saves lots (about 800k) objects into a shelve >> database (I'm using sqlite3dbm for this since all the default python dbm >> packages seem to be unreliable and effectively unusable, but t

Re: Memory usage steadily going up while pickling objects

2013-06-15 Thread Giorgos Tzampanakis
On 2013-06-15, Peter Otten wrote: > Giorgos Tzampanakis wrote: > >> I have a program that saves lots (about 800k) objects into a shelve >> database (I'm using sqlite3dbm for this since all the default python dbm >> packages seem to be unreliable and effectively unusable, but this is >> another dis

Re: Memory usage steadily going up while pickling objects

2013-06-14 Thread Peter Otten
Giorgos Tzampanakis wrote: > I have a program that saves lots (about 800k) objects into a shelve > database (I'm using sqlite3dbm for this since all the default python dbm > packages seem to be unreliable and effectively unusable, but this is > another discussion). > > The process takes about 10-

Re: Memory usage steadily going up while pickling objects

2013-06-14 Thread Dave Angel
On 06/14/2013 07:04 PM, Giorgos Tzampanakis wrote: I have a program that saves lots (about 800k) objects into a shelve database (I'm using sqlite3dbm for this since all the default python dbm packages seem to be unreliable and effectively unusable, but this is another discussion). The process ta

Memory usage steadily going up while pickling objects

2013-06-14 Thread Giorgos Tzampanakis
I have a program that saves lots (about 800k) objects into a shelve database (I'm using sqlite3dbm for this since all the default python dbm packages seem to be unreliable and effectively unusable, but this is another discussion). The process takes about 10-15 minutes. During that time I see memor