Re: Increasing maxMergedSegmentMB value

2016-01-31 Thread Zheng Lin Edwin Yeo
Hi Jack, Yes, I plan to merge all the 10GB segments to 20GB, and not just two of the segments. Sorry for the confusion. I have recently increased the system memory from 64GB to 192GB, but as our index size grows (which means more segments are created), I found that the query speed actually slow d

Re: Increasing maxMergedSegmentMB value

2016-01-31 Thread Jack Krupansky
Make sure you fully digest Mike McCandless' blog post on segment merge before trying to outguess his code: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html Generally, I don't think you would want to merge just two segments. Generally, you should do a bunch at a time,

Re: Increasing maxMergedSegmentMB value

2016-01-31 Thread Zheng Lin Edwin Yeo
Thanks for your reply Shawn and Jack. I wanted to increase the segment size to 15GB, so that there will be lesser segments to search for during the query, which should potentially improve the query speed. What if I set the segment size to 20GB? Will all the existing 10GB segments be merge to 20GB

Re: Increasing maxMergedSegmentMB value

2016-01-30 Thread Jack Krupansky
>From the Lucene MergePolicy Javadoc: "Whenever the segments in an index have been altered by IndexWriter , either the addition of a newly flushed segment, addition of many segments from addIndexes* calls, or a pre

Re: Increasing maxMergedSegmentMB value

2016-01-30 Thread Shawn Heisey
On 1/30/2016 7:31 AM, Zheng Lin Edwin Yeo wrote: > I would like to find out, when I increase the maxMergedSegmentMB from 10240 > (10GB) to 15360 (15GB), will all the 10GB segments that were created > previously be automatically merge to 15GB? Not necessarily. It will make those 10GB+ segments eli