Many good points! Thanks, guys! When background merge is employed, document additions can out-pace merging, no matter how many background merge threads are used. Blocking has to happen at some point.
So, if we do anything, we make it simple. I agree with what Robert and Yonik have proposed: documents can be buffered while segments are merged, but no more than maxBufferedDocs can be buffered at any time. Nadav asks a good question: if document additions can still block, what is the point? I see the following benefits: 1 Performance improvement to certain extent 2 Shorter ingestion hiccups: Especially when the ingestion rate is low, the hiccups can be very short. In the current single-threaded IndexWriter, the hiccups can be very long when a document ingestion triggers a cascade of merges. Doron has a valid concern: Lucene's being single threaded is a simplifying factor. Agree. That's why I propose this as a subclass of IndexWriter, not to replace the current single-threaded IndexWriter. So, given this simple background merge design, are people interested in the benefits it provides? --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
