On Thu, Jan 17, 2013 at 12:31:52PM -0500, Claudiu Saftoiu wrote: > I wrote the following code to preload the indices: > > def preload_index_btree(index_name, index_type, btree): > print "((Preloading '%s' %s index btree...))" % (index_name, > index_type) > start = last_print = time.time() > for i, item in enumerate(btree.items()): > item
That's a no-op: you might as well just write 'pass' here.
If you want to load the btree item into cache, you need to do
item._p_activate()
> print "((Preloaded '%s' %s index btree (%d items in %.2fs)))" % (
> index_name, index_type, i, time.time() - start,
> )
If you ever get an empty btree, you'll get an UnboundLocalError: 'i' here.
Drop the enumerate() trick and just use len(btree), it's efficient.
> def preload_catalog(catalog):
> """Given a catalog, touch every persistent object we can find to
> force
> them to go into the cache."""
> start = time.time()
> num_indices = len(catalog.items())
> for i, (index_name, index) in enumerate(catalog.items()):
> print "((Preloading index %2d/%2d '%s'...))" % (i+1,
> num_indices, index_name,)
> preload_index_btree(index_name, 'fwd', index._fwd_index)
> preload_index_btree(index_name, 'rev', index._rev_index)
> print "((Preloaded catalog! Took %.2fs))" % (time.time() - start)
>
> And I run it on server start as follows (modified for the relevant parts; I
> tried to make the example simple but it ended up needing a lot of parts).
> This runs in a thread:
>
> from util import zodb as Z
> from util import zodb_query as ZQ
> for i in xrange(3):
> connwrap = Z.ConnWrapper('index')
> print "((Preload #%d...))" % (i+1)
> with connwrap as index_root:
> ZQ.preload_catalog(index_root.index.catalog)
> connwrap.close()
Every thread has its own in-memory ZODB object cache, but if you have
configured a persistent ZEO client cache, it should help.
Marius Gedminas
--
Never trust a computer you can't repair yourself.
signature.asc
Description: Digital signature
_______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - [email protected] https://mail.zope.org/mailman/listinfo/zodb-dev
