Hi Shawn, Just checking to see if you saw my reply and had any feedback. Thank you again for your help. It is much appreciated. Thank you, Russ
From: Russell Bahr <r...@manzama.com> Date: Tuesday, October 15, 2019 at 11:50 AM To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> Subject: Re: solr 8.1.1 many time slower returning query results than solr 4.10.4 or solr 6.5.1 Hi Shawn, I included the wrong file for solr4 and did not realize until you pointed out the heap size. The correct file that is setting the Java environment is "Solr 4 tomcat setenv" I have uploaded that to the shared folder along with the requested screenshots "Solr 4 top screenshot","Solr 6 top screenshot","Solr 8 top screenshot". I have also uploaded the solr.log, solr_gc.log, and solr_slow_requests.log from a 2 hour period of time where I was running the email load test against the solr8 implementation in which the queued tasks are taking too long to complete. solr_gc.log, solr_gc.log.1, solr_gc.log.2, solr.log, solr.log.10, solr.log.6, solr.log.7, solr.log.8, solr.log.9, solr_slow_requests.log Let me know if there is any other information that I can provide that may help to work through this. Manzama a MODERN GOVERNANCE company Russell Bahr Lead Infrastructure Engineer USA & CAN Office: +1 (541) 306 3271 USA & CAN Support: +1 (541) 706 9393 UK Office & Support: +44 (0)203 282 1633 AUS Office & Support: +61 (0) 2 8417 2339 543 NW York Drive, Suite 100, Bend, OR 97703 LinkedIn<http://www.linkedin.com/company/manzama> | Twitter<https://twitter.com/ManzamaInc> | Facebook<http://www.facebook.com/manzamainc> | YouTube<https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> On Tue, Oct 15, 2019 at 2:28 AM Shawn Heisey <apa...@elyograg.org<mailto:apa...@elyograg.org>> wrote: On 10/14/2019 1:36 PM, Russell Bahr wrote: > Backend replacement of solr4 and hopefully Frontend replacement as well. > solr-spec 8.1.1 > lucene-spec 8.1.1 > Runtime Oracle Corporation OpenJDK 64-Bit Server VM 12 12+33 > 1 collection 6 shards 5 replicas per shard 17,919,889 current documents (35 > days worth of documents) - indexing new documents regularly throughout the > day, deleting aged out documents nightly. Java 12 is not recommended. It is one of the "new feature" releases that only gets 6 months of support. We would recommend Java 8 or Java 11. These are the versions with long term support. Probably a good thing to be using OpenJDK, as the official Oracle Java now requires paying for a license. Solr 8 ships with settings that enable the G1GC collector instead of CMS, because CMS is deprecated and will disappear in a future Java version. We have seen problems with this when the system is misconfigured as far as heap size. When the system is properly sized, G1 tends to do better than CMS, but when the heap is too large or too small, has a tendency to amplify garbage collection problems in comparison. Looking at your solr.in.sh<http://solr.in.sh> files for each version ... the Solr 4 install appears to be setting the heap to 512 megabytes. This is definitely not enough for millions of documents, and if this is what the heap size is actually set to, would almost certainly run into memory errors frequently and have absolutely terrible performance. But you are saying that it works well, so I don't think the heap is actually set to 512 megabytes. Maybe the bin/solr script has been modified directly to set the memory size instead of setting it in solr.in.sh<http://solr.in.sh> where it should be set. Solr 6 has a heap size of just under 27 gigabytes. Solr 8 has a heap size of just under 8 gigabytes. With millions of documents, it is likely that 8GB of heap is not quite big enough. For each of your installations (Solr 4, Solr 6, and Solr 8) can you provide the screenshot described at this wiki page? https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue It would also be helpful to see the GC logs from Solr 8. We would need at least one GC log, making sure that they cover at least a few hours, including the timeframe when the slow indexing and slow queries were observed. Thanks, Shawn
