On Thu, Oct 1, 2009 at 2:56 PM, Earwin Burrfoot <ear...@gmail.com> wrote: >>> If you're doing a full optimize, you're still using a single thread. Am I >>> wrong? >> >> Depends on how many merges are required, and, the merge scheduler. In >> this case (w/ 7000 segments, which is way too many, normally!), >> assuming ConcurrentMergeScheduler, multiple threads will be used since >> many merges will be pending. >> >> When it gets down to the last (enormous) merge, it's only one thread. > I'm speaking about full optimize. Is there any way to do it more > efficiently then running a single, last (enormous) merge?
I guess we could merge different parts of the index w/ different threads, if we wanted to push concurrency down into a single merge. > If you try to parallelize, you're merging some documents several times > (more work) and killing your disks, as merges are mostly IO-bound. Actually I've found merging of the postings to be CPU bound. I think the priority queue, and decode/encode of vInt, are the big CPU costs. Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
