Hi Aaron, You are right - large heap means that there will be no major GC all the time, but eventually it will happen and then the larger the heap the longer it will take. So with 300GB heap it takes observed 300s. If you used to run on 32GB heap and it was slow, it probably means that heap is too small for your load, but if you did not run into OOM, then it means it is starving but still can handle the load. In any case, I would not go with super large heap. One option is to split your server to more Solr instances. That would let you run more instances with smaller heap. I am not sure about your business case and if it is possible to do it without switching to SolrCloud, but if you have 100s of clients that are completely separate, you can simply split clients to several Solr instances that are running on the same server and have some logic that routes client to the right instance. If your average doc size is 5MB, then I would reduce the number of documents and that will reduce some load to heap. Unfortunately, the only way to answer similar questions is to run some tests.
Emir -- Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > On 18 Mar 2019, at 14:30, Aaron Yingcai Sun <y...@vizrt.com> wrote: > > I'm a bit confused, why large heap size would make it slower? Isn't that > give it enough room to make it not busy doing GC all the time? > > My http/json request contains 100 documents, the total size of the 100 > documents is around 5M, there are ~100 client sending those requests > continuously. > > Previously the JVM is set to max 32 GB , the speed was even worse, now it's > running with min 100GB, max 300GB, it use around 100GB. > > > this page suggest to use smaller number of documents per request, > https://wiki.apache.org/solr/SolrPerformanceProblems > > SolrPerformanceProblems - Solr > Wiki<https://wiki.apache.org/solr/SolrPerformanceProblems> > wiki.apache.org > General information. There is a performance bug that makes *everything* slow > in versions 6.4.0 and 6.4.1. The problem is fixed in 6.4.2. It is described > by SOLR-10130.This is highly version specific, so if you are not running one > of the affected versions, don't worry about it. > > So I try to reduce the number, still I could get lots of large response QTime: > > 190318-142652.695-160214 DBG1:doc_count: 10 , doc_size: 609 KB, Res code: > 200, QTime: 47918 ms, Request time: 47921 ms. > 190318-142652.704-160179 DBG1:doc_count: 10 , doc_size: 568 KB, Res code: > 200, QTime: 36919 ms, Request time: 36922 ms. > 190318-142652.780-160197 DBG1:doc_count: 10 , doc_size: 609 KB, Res code: > 200, QTime: 36082 ms, Request time: 36084 ms. > 190318-142652.859-160200 DBG1:doc_count: 10 , doc_size: 569 KB, Res code: > 200, QTime: 36880 ms, Request time: 36882 ms. > 190318-142653.131-160148 DBG1:doc_count: 10 , doc_size: 608 KB, Res code: > 200, QTime: 37222 ms, Request time: 37224 ms. > 190318-142653.154-160211 DBG1:doc_count: 10 , doc_size: 541 KB, Res code: > 200, QTime: 37241 ms, Request time: 37243 ms. > 190318-142653.223-163490 DBG1:doc_count: 10 , doc_size: 589 KB, Res code: > 200, QTime: 37174 ms, Request time: 37176 ms. > 190318-142653.359-160154 DBG1:doc_count: 10 , doc_size: 592 KB, Res code: > 200, QTime: 37008 ms, Request time: 37011 ms. > 190318-142653.497-163491 DBG1:doc_count: 10 , doc_size: 583 KB, Res code: > 200, QTime: 24828 ms, Request time: 24830 ms. > 190318-142653.987-160208 DBG1:doc_count: 10 , doc_size: 669 KB, Res code: > 200, QTime: 23900 ms, Request time: 23902 ms. > 190318-142654.114-160208 DBG1:doc_count: 10 , doc_size: 544 KB, Res code: > 200, QTime: 121 ms, Request time: 122 ms. > 190318-142654.233-160208 DBG1:doc_count: 10 , doc_size: 536 KB, Res code: > 200, QTime: 113 ms, Request time: 115 ms. > 190318-142654.354-160208 DBG1:doc_count: 10 , doc_size: 598 KB, Res code: > 200, QTime: 116 ms, Request time: 117 ms. > 190318-142654.466-160208 DBG1:doc_count: 10 , doc_size: 546 KB, Res code: > 200, QTime: 107 ms, Request time: 108 ms. > 190318-142654.586-160208 DBG1:doc_count: 10 , doc_size: 566 KB, Res code: > 200, QTime: 114 ms, Request time: 115 ms. > 190318-142654.687-160208 DBG1:doc_count: 10 , doc_size: 541 KB, Res code: > 200, QTime: 96 ms, Request time: 98 ms. > 190318-142654.768-160208 DBG1:doc_count: 10 , doc_size: 455 KB, Res code: > 200, QTime: 75 ms, Request time: 77 ms. > 190318-142654.870-160208 DBG1:doc_count: 10 , doc_size: 538 KB, Res code: > 200, QTime: 97 ms, Request time: 98 ms. > 190318-142654.967-160208 DBG1:doc_count: 10 , doc_size: 539 KB, Res code: > 200, QTime: 92 ms, Request time: 93 ms. > 190318-142655.096-160208 DBG1:doc_count: 10 , doc_size: 672 KB, Res code: > 200, QTime: 124 ms, Request time: 125 ms. > 190318-142655.210-160208 DBG1:doc_count: 10 , doc_size: 605 KB, Res code: > 200, QTime: 108 ms, Request time: 110 ms. > 190318-142655.304-160208 DBG1:doc_count: 10 , doc_size: 481 KB, Res code: > 200, QTime: 89 ms, Request time: 90 ms. > 190318-142655.410-160208 DBG1:doc_count: 10 , doc_size: 468 KB, Res code: > 200, QTime: 101 ms, Request time: 102 ms. > > > ________________________________ > From: Toke Eskildsen <t...@kb.dk> > Sent: Monday, March 18, 2019 2:13:12 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr index slow response > > On Mon, 2019-03-18 at 10:47 +0000, Aaron Yingcai Sun wrote: >> Solr server is running on a quit powerful server, 32 cpus, 400GB RAM, >> while 300 GB is reserved for solr, [...] > > 300GB for Solr sounds excessive. > >> Our application send 100 documents to solr per request, json encoded. >> the size is around 5M each time. some times the response time is >> under 1 seconds, some times could be 300 seconds, the slow response >> happens very often. >> ... >> There are around 100 clients sending those documents at the same >> time, but each for the client is blocking call which wait the http >> response then send the next one. > > 100 clients * 5MB/batch = at most 500MB. Or maybe you meant 100 clients > * 100 documents * 5MB/document = at most 50GB? Either way it is a long > way from 300GB and the stats you provide further down the thread > indicates that you are overprovisioning quite a lot: > > "memory":{ > "free":"69.1 GB", > "total":"180.2 GB", > "max":"266.7 GB", > "used":"111 GB (%41.6)", > > Intermittent slow response times are a known effect of having large > Java heaps, due to stop-the-world garbage collections. > > Try dialing Xmx _way_ down: If your batches are only 5MB each, try > Xmx=20g or less. I know that the stats above says that Solr uses 111GB, > but the JVM has a tendency to expand the heap quite a lot when it is > getting hammered. If you want to check beforehand, you can see how much > memeory is freed from full GCs in the GC-log. > > - Toke Eskildsen, Royal Danish Library > >
