Thanks for replying! it is indeed the same Java (OpenJDK 11) we're using for both. I tried upgrading to 14 and the heap *does* stay a little smaller, but we still see the large (>10sec) GC durations happening.
I've not been able to get switching to CMS to work, unfortunately On Thu, 2 Sept 2021 at 16:52, Shawn Heisey <[email protected]> wrote: > On 9/2/2021 9:28 AM, Dominic Humphries wrote: > > We're trying to upgrade from 8.3.1 to 8.8.1 but my pre-release testing > has > > shown us some performance issues. Examination of the GC log shows that > the > > possible cause may be here: > > > > 8.3.1 graphs: https://imgur.com/a/ZM9wdob > > 8.8.1 graphs: https://imgur.com/a/UzMinwJ > > > > The test cycle here is 2 mins with requests; 2 mins no requests; 2 mins > > requests. You can see the 8.3 gives what I'd expect - fairly consistent > > heap usage, fairly fast & consistent GC durations. > > 8.8 however shows steadily increasing heap usage in the first request > cycle > > and a big spike in one of the GC durations. > > With such a small heap, 12 seconds for a GC seems extremely excessive. > It simply shouldn't take that long for a heap that size. If your heap > were 8GB or more, then I could understand that happening. The general > solution when there are full GCs is to increase the heap size so that > Java is more likely to choose the faster generation-specific collections > instead of full GC. > > My recommendation would be to upgrade or change your Java version. If > you're about to tell me that it's the same Java version in both cases, > there could be differences in how the two versions work that cause the > newer one to trigger a bug in Java's GC that doesn't happen with the > older version. The latest Java 8 should be OK. If you want to go with > a later Java version, whether or not that will work will depend on how > old your Solr version is. I would stick with the latest releases of the > LTS Java versions -- 8, 11, 14, and 17 when it gets released. Previously > I would have recommended Oracle Java, but they changed their licensing > so that most people must pay for it, so it would be better to go with > one of the free alternatives. OpenJDK would be a great option. Avoid > IBM's Java or any vendor that inherits from it -- IBM's Java causes > known problems. > > You could try changing the GC to CMS, instead of the default of G1 that > Solr now ships with. > > > https://cwiki.apache.org/confluence/display/solr/shawnheisey#ShawnHeisey-CMS(ConcurrentMarkSweep)Collector > > That wiki page shows the environment variable as JVM_OPTS, because when > I wrote it I had my own init script -- at the time, Solr didn't have > one. I believe that to use it in solr.in.sh or solr.in.cmd you would > need to change that to GC_TUNE instead. I will get around to changing > it to reflect modern Solr versions. > > Thanks, > Shawn > >
