I believe it is common to have a theshold (say every 10000) objects to do a savepoint and then you can run cacheMinimize after the savepoint.
surprised the zc.catalog does not provide this sort of knob. maybe it was a problem in the original ZCatalog implementation? cheers alan On Wed, Jul 30, 2008 at 1:21 PM, <[EMAIL PROTECTED]> wrote: > > I have a database consisting mainly of an IOBTree of ~700,000 items > (each persistent mappings), and an zc.catalog indexing them by one of the > mapping's fields (a date). I am opening the FileStorage in read_only mode. > For each day in the index, I get the day's mappings and read the contents > of another field. As I go through each day, the memory usage explodes (over > 32 GB). Is there a way to configure the cache to automatically keep itself > under the value of the cache size parameter? > > To avoid this problem in most cases, I wrap the IOBTree in another > object which does nothing more than call db.cacheMinimize after every 10000 > items are iterated over. But for random access, that's not an option. > > From reading the archives, it sounds like cache cleaning does not > happen while running a transaction. Is that my problem? I'm in read-only > mode, so I can't perform a transaction anyway, but could the ZODB think I > want to? Or is my problem something else? > > Maybe if someone can point me to a description of how the caching > works. > > -- > Anthony Foglia > Princeton Consultants > (609) 987-8787 x233 > _______________________________________________ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - [email protected] > http://mail.zope.org/mailman/listinfo/zodb-dev > > -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 _______________________________________________ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - [email protected] http://mail.zope.org/mailman/listinfo/zodb-dev
