I'm the primary zc.catalog author.
He is running read-only, so savepoints shouldn't be very pertinent.
The original poster is correct that the cache is cleaned out at
transaction boundaries (or savepoints, as Alan said).
I don't have experience with running the db in readonly mode in
production. That said, if you have obvious boundaries for your
searches, I suspect that beginning a transaction at the boundary would
probably do the trick, and also work (with invalidations and the like)
if you ever have a readonly connection to a ZEO database server that
is getting updates from another source.
While it's interesting to contemplate the knob that Alan mentioned,
I'm reasonably confident that this logic belongs in the app, not the
index or catalog.
Gary
On Jul 31, 2008, at 11:55 AM, Alan Runyan wrote:
I believe it is common to have a theshold (say every 10000) objects to
do a savepoint
and then you can run cacheMinimize after the savepoint.
surprised the zc.catalog does not provide this sort of knob. maybe it
was a problem
in the original ZCatalog implementation?
cheers
alan
On Wed, Jul 30, 2008 at 1:21 PM, <[EMAIL PROTECTED]> wrote:
I have a database consisting mainly of an IOBTree of
~700,000 items
(each persistent mappings), and an zc.catalog indexing them by one
of the
mapping's fields (a date). I am opening the FileStorage in
read_only mode.
For each day in the index, I get the day's mappings and read the
contents
of another field. As I go through each day, the memory usage
explodes (over
32 GB). Is there a way to configure the cache to automatically
keep itself
under the value of the cache size parameter?
To avoid this problem in most cases, I wrap the IOBTree in
another
object which does nothing more than call db.cacheMinimize after
every 10000
items are iterated over. But for random access, that's not an
option.
From reading the archives, it sounds like cache cleaning
does not
happen while running a transaction. Is that my problem? I'm in
read-only
mode, so I can't perform a transaction anyway, but could the ZODB
think I
want to? Or is my problem something else?
Maybe if someone can point me to a description of how the
caching
works.
--
Anthony Foglia
Princeton Consultants
(609) 987-8787 x233
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - [email protected]
http://mail.zope.org/mailman/listinfo/zodb-dev
--
Alan Runyan
Enfold Systems, Inc.
http://www.enfoldsystems.com/
phone: +1.713.942.2377x111
fax: +1.832.201.8856
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - [email protected]
http://mail.zope.org/mailman/listinfo/zodb-dev
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - [email protected]
http://mail.zope.org/mailman/listinfo/zodb-dev