> One machine runs with a 3TB drive, running 3 solr processes (each with one core as described above).
How much total memory on the machine? Kevin Risden On Sat, Nov 11, 2017 at 1:08 PM, Nawab Zada Asad Iqbal <khi...@gmail.com> wrote: > Thanks for a quick and detailed response, Erick! > > Unfortunately i don't have a proof, but our servers with solr 4.5 are > running really nicely with the above config. I had assumed that same or > similar settings will also perform well with Solr 7, but that assumption > didn't hold. As, a lot has changed in 3 major releases. > I have tweaked the cache values as you suggested but increasing or > decreasing doesn't seem to do any noticeable improvement. > > At the moment, my one core has 800GB index, ~450 Million documents, 48 G > Xmx. GC pauses haven't been an issue though. One machine runs with a 3TB > drive, running 3 solr processes (each with one core as described above). I > agree that it is a very atypical system so i should probably try different > parameters with a fresh eye to find the solution. > > > I tried with autocommits (commit with opensearcher=false very half minute ; > and softcommit every 5 minutes). That supported the hypothesis that the > query throughput decreases after opening a new searcher and **not** after > committing the index . Cache hit ratios are all in 80+% (even when i > decreased the filterCache to 128, so i will keep it at this lower value). > Document cache hitratio is really bad, it drops to around 40% after > newSearcher. But i guess that is expected, since it cannot be warmed up > anyway. > > > Thanks > Nawab > > > > On Thu, Nov 9, 2017 at 9:11 PM, Erick Erickson <erickerick...@gmail.com> > wrote: > > > What evidence to you have that the changes you've made to your configs > > are useful? There's lots of things in here that are suspect: > > > > <double name="forceMergeDeletesPctAllowed">1</double> > > > > First, this is useless unless you are forceMerging/optimizing. Which > > you shouldn't be doing under most circumstances. And you're going to > > be rewriting a lot of data every time See: > > > > https://lucidworks.com/2017/10/13/segment-merging-deleted- > > documents-optimize-may-bad/ > > > > filterCache size of size="10240" is far in excess of what we usually > > recommend. Each entry can be up to maxDoc/8 and you have 10K of them. > > Why did you choose this? On the theory that "more is better?" If > > you're using NOW then you may not be using the filterCache well, see: > > > > https://lucidworks.com/2012/02/23/date-math-now-and-filter-queries/ > > > > autowarmCount="1024" > > > > Every time you commit you're firing off 1024 queries which is going to > > spike the CPU a lot. Again, this is super-excessive. I usually start > > with 16 or so. > > > > Why are you committing from a cron job? Why not just set your > > autocommit settings and forget about it? That's what they're for. > > > > Your queryResultCache is likewise kind of large, but it takes up much > > less space than the filterCache per entry so it's probably OK. I'd > > still shrink it and set the autowarm to 16 or so to start, unless > > you're seeing a pretty high hit ratio, which is pretty unusual but > > does happen. > > > > 48G of memory is just asking for long GC pauses. How many docs do you > > have in each core anyway? If you're really using this much heap, then > > it'd be good to see what you can do to shrink in. Enabling docValues > > for all fields you facet, sort or group on will help that a lot if you > > haven't already. > > > > How much memory on your entire machine? And how much is used by _all_ > > the JVMs you running on a particular machine? MMapDirectory needs as > > much OS memory space as it can get, see: > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > > > Lately we've seen some structures that consume memory until a commit > > happens (either soft or hard). I'd shrink my autocommit down to 60 > > seconds or even less (openSearcher=false). > > > > In short, I'd go back mostly to the default settings and build _up_ as > > you can demonstrate improvements. You've changed enough things here > > that untangling which one is the culprit will be hard. You want the > > JVM to have as little memory as possible, unfortunately that's > > something you figure out by experimentation. > > > > Best, > > Erick > > > > On Thu, Nov 9, 2017 at 8:42 PM, Nawab Zada Asad Iqbal <khi...@gmail.com> > > wrote: > > > Hi, > > > > > > I am committing every 5 minutes using a periodic cron job "curl > > > http://localhost:8984/solr/core1/update?commit=true";. Besides this, my > > app > > > doesn't do any soft or hard commits. With Solr 7 upgrade, I am noticing > > > that query throughput plummets every 5 minutes - probably when the > commit > > > happens. > > > What can I do to improve this? I didn't use to happen like this in > > solr4.5. > > > (i.e., i used to get a stable query throughput of 50-60 queries per > > > second. Now there are spikes to 60 qps interleaved by drops to almost > > > **0**). Between those 5 minutes, I am able to achieve high throughput, > > > hence I guess that issue is related to indexing or merging, and not > query > > > flow. > > > > > > I have 48G allotted to each solr process, and it seems that only ~50% > is > > > being used at any time, similarly CPU is not spiking beyond 50% either. > > > There is frequent merging (every 5 minute) , but i am not sure if that > is > > > a cause of the slowdown. > > > > > > Here are my merge and cache settings: > > > > > > Thanks > > > Nawab > > > > > > <mergePolicyFactory class="org.apache.solr.index. > > TieredMergePolicyFactory"> > > > <int name="maxMergeAtOnce">5</int> > > > <int name="segmentsPerTier">5</int> > > > <int name="maxMergeAtOnceExplicit">10</int> > > > <int name="floorSegmentMB">16</int> > > > <!-- 50 gb --> > > > <double name="maxMergedSegmentMB">50000</double> > > > <double name="forceMergeDeletesPctAllowed">1</double> > > > > > > </mergePolicyFactory> > > > > > > > > > > > > > > > <filterCache class="solr.FastLRUCache" > > > size="10240" > > > initialSize="5120" > > > autowarmCount="1024"/> > > > <queryResultCache class="solr.LRUCache" > > > size="10240" > > > initialSize="5120" > > > autowarmCount="0"/> > > > <documentCache class="solr.LRUCache" > > > size="10240" > > > initialSize="5120" > > > autowarmCount="0"/> > > > > > > > > > <useColdSearcher>false</useColdSearcher> > > > > > > <maxWarmingSearchers>2</maxWarmingSearchers> > > > > > > <listener event="newSearcher" class="solr.QuerySenderListener"> > > > <arr name="queries"> > > > </arr> > > > </listener> > > > <listener event="firstSearcher" class="solr.QuerySenderListener"> > > > <arr name="queries"> > > > </arr> > > > </listener> > > >
