On Mon, 2007-09-17 at 19:57 -0400, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Roché Compaan wrote: > > Hi there > > > > Are the current indexing solutions that we have in Zope 2 and 3 the best > > that we can have? Is there any way to improve the amount of concurrent > > indexing a field or text index can handle. The Zope 3 implementation > > doesn't look significantly different to the Zope 2 one in that it still > > uses a BTree for forward and reverse index. > > > > I have a basic stress test where 10 concurrent threads index 10 random > > words in a FieldIndex and get too many conflict errors to even consider > > this a usable scalable indexing solution. I don't really want to index > > objects in another backend. I really would like to make the ZODB work > > for me here. > > > > Are there solutions here? Can one employ some of the QueueCatalog > > conflict resolution strategies to make indexes more resilient? Or should > > one use some locking strategy instead? > > Check out QueueCatalog, which batches up indexing changes for processing > within a single thread. > > svn://svn.zope.org/repos/main/Products.QueueCatalog
I use QueueCatalog often and I know how it works. But if an application requires immediate indexing then QueueCatalog is not a solution. Sorry if I was unclear but what I'm really asking is if it is possible to improve the conflict handling of the current indexes that we have in Zope. I am also asking if concurrent indexing in the ZODB is a realistic goal. The reason I mentioned QueueCatalog is not because of its batch indexing in one thread, but because it has a lot of conflict handling on the queue itself. Sessions might be another good example of a product that tries hard to handle conflicts. Do these products have strategies that can be made to work for indexes? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )