On Thu, Jan 17, 2013 at 12:31:52PM -0500, Claudiu Saftoiu wrote: > I wrote the following code to preload the indices: > > def preload_index_btree(index_name, index_type, btree): > print "((Preloading '%s' %s index btree...))" % (index_name, > index_type) > start = last_print = time.time() > for i, item in enumerate(btree.items()): > item
That's a no-op: you might as well just write 'pass' here. If you want to load the btree item into cache, you need to do item._p_activate() > print "((Preloaded '%s' %s index btree (%d items in %.2fs)))" % ( > index_name, index_type, i, time.time() - start, > ) If you ever get an empty btree, you'll get an UnboundLocalError: 'i' here. Drop the enumerate() trick and just use len(btree), it's efficient. > def preload_catalog(catalog): > """Given a catalog, touch every persistent object we can find to > force > them to go into the cache.""" > start = time.time() > num_indices = len(catalog.items()) > for i, (index_name, index) in enumerate(catalog.items()): > print "((Preloading index %2d/%2d '%s'...))" % (i+1, > num_indices, index_name,) > preload_index_btree(index_name, 'fwd', index._fwd_index) > preload_index_btree(index_name, 'rev', index._rev_index) > print "((Preloaded catalog! Took %.2fs))" % (time.time() - start) > > And I run it on server start as follows (modified for the relevant parts; I > tried to make the example simple but it ended up needing a lot of parts). > This runs in a thread: > > from util import zodb as Z > from util import zodb_query as ZQ > for i in xrange(3): > connwrap = Z.ConnWrapper('index') > print "((Preload #%d...))" % (i+1) > with connwrap as index_root: > ZQ.preload_catalog(index_root.index.catalog) > connwrap.close() Every thread has its own in-memory ZODB object cache, but if you have configured a persistent ZEO client cache, it should help. Marius Gedminas -- Never trust a computer you can't repair yourself.
signature.asc
Description: Digital signature
_______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev