I think it'd be interesting to see how much better this is and in what kinds of scenarios it makes things faster, so if this is going to be separate from IndexWriter, why not try it.
Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share ----- Original Message ---- From: Ning Li <[EMAIL PROTECTED]> To: java-dev@lucene.apache.org Sent: Friday, March 2, 2007 5:44:55 PM Subject: Re: Concurrent merge Many good points! Thanks, guys! When background merge is employed, document additions can out-pace merging, no matter how many background merge threads are used. Blocking has to happen at some point. So, if we do anything, we make it simple. I agree with what Robert and Yonik have proposed: documents can be buffered while segments are merged, but no more than maxBufferedDocs can be buffered at any time. Nadav asks a good question: if document additions can still block, what is the point? I see the following benefits: 1 Performance improvement to certain extent 2 Shorter ingestion hiccups: Especially when the ingestion rate is low, the hiccups can be very short. In the current single-threaded IndexWriter, the hiccups can be very long when a document ingestion triggers a cascade of merges. Doron has a valid concern: Lucene's being single threaded is a simplifying factor. Agree. That's why I propose this as a subclass of IndexWriter, not to replace the current single-threaded IndexWriter. So, given this simple background merge design, are people interested in the benefits it provides? --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
