Hi Shawn, Still hoping for some feedback here. Should I not be replying to this thread and instead create a new one? As I do not see an improvement when using java11 I am now going to rebuild again with java8 and solr 8.1.1. Please respond and let me know if I am going in the right direction, or should be attacking this in a different way. Thank you, Russ
*Manzama*a MODERN GOVERNANCE company Russell Bahr Lead Infrastructure Engineer USA & CAN Office: +1 (541) 306 3271 USA & CAN Support: +1 (541) 706 9393 UK Office & Support: +44 (0)203 282 1633 AUS Office & Support: +61 (0) 2 8417 2339 543 NW York Drive, Suite 100, Bend, OR 97703 LinkedIn <http://www.linkedin.com/company/manzama> | Twitter <https://twitter.com/ManzamaInc> | Facebook <http://www.facebook.com/manzamainc> | YouTube <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> On Tue, Oct 22, 2019 at 1:18 PM Russell Bahr <r...@manzama.com> wrote: > Hi, > Is there anyone that would be able to assist with the issue that I am > seeing? > I am seeing the same slowness with solr 8.1.1 using java11 as I am seeing > with java12, over queries that are run from solr4.10.4 with java8 and > solr6.5.1 with java8. > Queries that return in less than half a second on solr4 are taking up to > 20 seconds with same data indexed to solr 8.1.1 > I have posted configs, schemas, and various log files in this shared > dropbox folder ( > https://www.dropbox.com/sh/2x2k5c9db7d4pt9/AADnHwuJc7a9Fh4KmUD15rS0a?dl=0 > ). > Any additional help/assistance would be greatly appreciated. > Thank you, > Russ > > *Manzama*a MODERN GOVERNANCE company > > Russell Bahr > Lead Infrastructure Engineer > > USA & CAN Office: +1 (541) 306 3271 > USA & CAN Support: +1 (541) 706 9393 > UK Office & Support: +44 (0)203 282 1633 > AUS Office & Support: +61 (0) 2 8417 2339 > > 543 NW York Drive, Suite 100, Bend, OR 97703 > > LinkedIn <http://www.linkedin.com/company/manzama> | Twitter > <https://twitter.com/ManzamaInc> | Facebook > <http://www.facebook.com/manzamainc> | YouTube > <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> > > > On Sun, Oct 20, 2019 at 8:54 PM Russell Bahr <r...@manzama.com> wrote: > >> Hi Shawn, >> per your comments from before >> On Oct 15, 2019, 2:28 AM, Shawn Heisey wrote: >> > Java 12 is not recommended. It is one of the "new feature" releases >> > that only gets 6 months of support. We would recommend Java 8 or Java >> > 11. These are the versions with long term support. Probably a good >> > thing to be using OpenJDK, as the official Oracle Java now requires >> > paying for a license. >> >> I have rebuilt my 30 server sorl 8 cluster using java11 and increased the >> java heap -Xms10433m -Xmx10433m and am seeing the same slowness that I >> was seeing with java12. >> I have not yet tried to build out the solr 8 collection with java8. >> Would it be worthwhile to do that or were you able to see anything in the >> logs? >> >> Thank you in advance, >> Russ >> >> *Manzama*a MODERN GOVERNANCE company >> >> Russell Bahr >> Lead Infrastructure Engineer >> >> USA & CAN Office: +1 (541) 306 3271 >> USA & CAN Support: +1 (541) 706 9393 >> UK Office & Support: +44 (0)203 282 1633 >> AUS Office & Support: +61 (0) 2 8417 2339 >> >> 543 NW York Drive, Suite 100, Bend, OR 97703 >> >> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter >> <https://twitter.com/ManzamaInc> | Facebook >> <http://www.facebook.com/manzamainc> | YouTube >> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> >> >> >> On Wed, Oct 16, 2019 at 11:50 AM Russell Bahr <r...@manzama.com> wrote: >> >>> Hi Shawn, >>> >>> Just checking to see if you saw my reply and had any feedback. Thank you >>> again for your help. It is much appreciated. >>> >>> Thank you, >>> >>> Russ >>> >>> >>> >>> >>> >>> *From: *Russell Bahr <r...@manzama.com> >>> *Date: *Tuesday, October 15, 2019 at 11:50 AM >>> *To: *"solr-user@lucene.apache.org" <solr-user@lucene.apache.org> >>> *Subject: *Re: solr 8.1.1 many time slower returning query results than >>> solr 4.10.4 or solr 6.5.1 >>> >>> >>> >>> Hi Shawn, >>> >>> I included the wrong file for solr4 and did not realize until you >>> pointed out the heap size. The correct file that is setting the Java >>> environment is "Solr 4 tomcat setenv" I have uploaded that to the shared >>> folder along with the requested screenshots "Solr 4 top screenshot","Solr 6 >>> top screenshot","Solr 8 top screenshot". >>> >>> >>> >>> I have also uploaded the solr.log, solr_gc.log, and >>> solr_slow_requests.log from a 2 hour period of time where I was running the >>> email load test against the solr8 implementation in which the queued tasks >>> are taking too long to complete. >>> >>> >>> >>> solr_gc.log, solr_gc.log.1, solr_gc.log.2, solr.log, solr.log.10, >>> solr.log.6, solr.log.7, solr.log.8, solr.log.9, solr_slow_requests.log >>> >>> >>> >>> Let me know if there is any other information that I can provide that >>> may help to work through this. >>> >>> >>> >>> *Manzama *a MODERN GOVERNANCE company >>> >>> >>> >>> Russell Bahr >>> Lead Infrastructure Engineer >>> >>> USA & CAN Office: +1 (541) 306 3271 >>> USA & CAN Support: +1 (541) 706 9393 >>> UK Office & Support: +44 (0)203 282 1633 >>> AUS Office & Support: +61 (0) 2 8417 2339 >>> >>> 543 NW York Drive, Suite 100, Bend, OR 97703 >>> >>> LinkedIn <http://www.linkedin.com/company/manzama> | Twitter >>> <https://twitter.com/ManzamaInc> | Facebook >>> <http://www.facebook.com/manzamainc> | YouTube >>> <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> >>> >>> >>> >>> >>> >>> On Tue, Oct 15, 2019 at 2:28 AM Shawn Heisey <apa...@elyograg.org> >>> wrote: >>> >>> On 10/14/2019 1:36 PM, Russell Bahr wrote: >>> > Backend replacement of solr4 and hopefully Frontend replacement as >>> well. >>> > solr-spec 8.1.1 >>> > lucene-spec 8.1.1 >>> > Runtime Oracle Corporation OpenJDK 64-Bit Server VM 12 12+33 >>> > 1 collection 6 shards 5 replicas per shard 17,919,889 current >>> documents (35 days worth of documents) - indexing new documents regularly >>> throughout the day, deleting aged out documents nightly. >>> >>> Java 12 is not recommended. It is one of the "new feature" releases >>> that only gets 6 months of support. We would recommend Java 8 or Java >>> 11. These are the versions with long term support. Probably a good >>> thing to be using OpenJDK, as the official Oracle Java now requires >>> paying for a license. >>> >>> Solr 8 ships with settings that enable the G1GC collector instead of >>> CMS, because CMS is deprecated and will disappear in a future Java >>> version. We have seen problems with this when the system is >>> misconfigured as far as heap size. When the system is properly sized, >>> G1 tends to do better than CMS, but when the heap is too large or too >>> small, has a tendency to amplify garbage collection problems in >>> comparison. >>> >>> Looking at your solr.in.sh files for each version ... the Solr 4 >>> install >>> appears to be setting the heap to 512 megabytes. This is definitely not >>> enough for millions of documents, and if this is what the heap size is >>> actually set to, would almost certainly run into memory errors >>> frequently and have absolutely terrible performance. But you are saying >>> that it works well, so I don't think the heap is actually set to 512 >>> megabytes. Maybe the bin/solr script has been modified directly to set >>> the memory size instead of setting it in solr.in.sh where it should be >>> set. >>> >>> Solr 6 has a heap size of just under 27 gigabytes. Solr 8 has a heap >>> size of just under 8 gigabytes. With millions of documents, it is >>> likely that 8GB of heap is not quite big enough. >>> >>> For each of your installations (Solr 4, Solr 6, and Solr 8) can you >>> provide the screenshot described at this wiki page? >>> >>> >>> https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue >>> >>> It would also be helpful to see the GC logs from Solr 8. We would need >>> at least one GC log, making sure that they cover at least a few hours, >>> including the timeframe when the slow indexing and slow queries were >>> observed. >>> >>> Thanks, >>> Shawn >>> >>>
