On 9/2/2021 9:28 AM, Dominic Humphries wrote:
We're trying to upgrade from 8.3.1 to 8.8.1 but my pre-release testing has
shown us some performance issues. Examination of the GC log shows that the
possible cause may be here:
8.3.1 graphs: https://imgur.com/a/ZM9wdob
8.8.1 graphs: https://imgur.com/a/UzMinwJ
The test cycle here is 2 mins with requests; 2 mins no requests; 2 mins
requests. You can see the 8.3 gives what I'd expect - fairly consistent
heap usage, fairly fast & consistent GC durations.
8.8 however shows steadily increasing heap usage in the first request cycle
and a big spike in one of the GC durations.
With such a small heap, 12 seconds for a GC seems extremely excessive.
It simply shouldn't take that long for a heap that size. If your heap
were 8GB or more, then I could understand that happening. The general
solution when there are full GCs is to increase the heap size so that
Java is more likely to choose the faster generation-specific collections
instead of full GC.
My recommendation would be to upgrade or change your Java version. If
you're about to tell me that it's the same Java version in both cases,
there could be differences in how the two versions work that cause the
newer one to trigger a bug in Java's GC that doesn't happen with the
older version. The latest Java 8 should be OK. If you want to go with
a later Java version, whether or not that will work will depend on how
old your Solr version is. I would stick with the latest releases of the
LTS Java versions -- 8, 11, 14, and 17 when it gets released. Previously
I would have recommended Oracle Java, but they changed their licensing
so that most people must pay for it, so it would be better to go with
one of the free alternatives. OpenJDK would be a great option. Avoid
IBM's Java or any vendor that inherits from it -- IBM's Java causes
known problems.
You could try changing the GC to CMS, instead of the default of G1 that
Solr now ships with.
https://cwiki.apache.org/confluence/display/solr/shawnheisey#ShawnHeisey-CMS(ConcurrentMarkSweep)Collector
That wiki page shows the environment variable as JVM_OPTS, because when
I wrote it I had my own init script -- at the time, Solr didn't have
one. I believe that to use it in solr.in.sh or solr.in.cmd you would
need to change that to GC_TUNE instead. I will get around to changing
it to reflect modern Solr versions.
Thanks,
Shawn
