I agree that the current blocking model works for some applications, especially if the indexes are batch built.
But other applications, e.g. with online indexes, would greatly benefit from a non-blocking model. Most systems that merge data support background merges. As long as we keep it simple (how about the original proposal?), applications will benefit from this. Ning On 2/20/07, Mike Klaas <[EMAIL PROTECTED]> wrote:
On 2/20/07, robert engels <[EMAIL PROTECTED]> wrote: > What about a queue of segments to merge. The add document will add > segments to the queue, if the queue contains too many segments it > blocks. > > Another thread reads the segments from the queue and merges them. > > This would effectively block adding of documents some times, but that > is not different than what happens now. In our custom database (that has a lucene-like logarithmic set of segmented databases), we have a separate merge thread controlled with a bounded queue as you suggest. As you note, the unbounded wait time on doc addition must still be present if you have limits on how far ahead you can let the doc addition run ahead of the merging. You could imagine more complexity, allowing smaller sub-merges to occur of fresh segments while large background merges were proceeding. For us, ironing out the resulting complexity wasn't a beneficial use of time, and in retrospect I prefer lucene's model. -Mike --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
