Shawn,Erick, Thank you for the explanation. The merge scheduler params make sense now.
Thanks, Rahul On Wed, Jul 3, 2019 at 11:30 AM Erick Erickson <erickerick...@gmail.com> wrote: > Two more tidbits to add to Shawn’s explanation: > > There are heuristics built in to ConcurrentMergeScheduler. > From the Javadocs: > * If it's an SSD, > * {@code maxThreadCount} is set to {@code max(1, min(4, cpuCoreCount/2))}, > * otherwise 1. Note that detection only currently works on > * Linux; other platforms will assume the index is not on an SSD. > > Second, TieredMergePolicy (the default) merges in “tiers” that > are of similar size. So you can have multiple merges going on > at the same time on disjoint sets of segments. > > Best, > Erick > > > On Jul 3, 2019, at 7:54 AM, Shawn Heisey <apa...@elyograg.org> wrote: > > > > On 7/2/2019 10:53 PM, Rahul Goswami wrote: > >> Hi Shawn, > >> Thank you for the detailed suggestions. Although, I would like to > >> understand the maxMergeCount and maxThreadCount params better. The > >> documentation > >> < > https://lucene.apache.org/solr/guide/7_3/indexconfig-in-solrconfig.html#mergescheduler > > > >> mentions > >> that > >> maxMergeCount : The maximum number of simultaneous merges that are > allowed. > >> maxThreadCount : The maximum number of simultaneous merge threads that > >> should be running at once > >> Since one thread can only do 1 merge at any given point of time, how > does > >> maxMergeCount being greater than maxThreadCount help anyway? I am having > >> difficulty wrapping my head around this, and would appreciate if you > could > >> help clear it for me. > > > > The maxMergeCount setting controls the number of merges that can be > *scheduled* at the same time. As soon as that number of merges is reached, > the indexing thread(s) will be paused until the number of merges in the > schedule drops below this number. This ensures that no more merges will be > scheduled. > > > > By setting maxMergeCount higher than the number of merges that are > expected in the schedule, you can ensure that indexing will never be > paused. It would require very atypical merge policy settings for the > number of scheduled merges to ever reach six. On my own indexing, I > reached three scheduled merges quite frequently. The default setting for > maxMergeCount is three. > > > > The maxThreadCount setting controls how many of the scheduled merges > will be simultaneously executed. With index data on standard spinning > disks, you do not want to increase this number beyond 1, or you will have a > performance problem due to thrashing disk heads. If your data is on SSD, > you can make it larger than 1. > > > > Thanks, > > Shawn > >