Awesome, very helpful. Do you maybe want to add this to the Solr wiki
somewhere? Finding some advice for JVM tuning for Solr can be
challenging, and you've explained what you did and why very well.
On 3/17/2011 2:59 PM, Dyer, James wrote:
We're on the final stretch in getting our product database in Production with Solr. We have 13m
"wide-ish" records with quite a few stored fields in a single index (no shards). We sort on at
least a dozen fields and facet on 20-30. One thing that came up in QA testing is we were getting full gc's
due to "promotion failed" conditions. This led us to believe we were dealing with large objects
being created and a fragmented old generation. After improving, but not solving, the problem by tweaking
"conventional" jvm parameters, our JVM expert learned about some newer tuning params included in
Sun/Oracle's JDK 1.6.0_24 (we're running RHEL x64, but I think these are available on other platforms too):
These 3 options dramatically reduced the # objects getting promoted into the Old
Gen, reducing fragmentation and CMS frequency& time:
-XX:+UseStringCache
-XX:+OptimizeStringConcat
-XX:+UseCompressedStrings
This uses compressed pointers on a 64-bit JVM, significantly reducing the
memory& performance penalty in using a 64-bit jvm over 32-bit. This reduced
our new GC (ParNew) time significantly:
-XX:+UseCompressedOops
The default for this was causing CMS to begin too late sometimes. (the
documentated 68% proved false in our case. We figured it was defaulting close
to 90%) Much lower than 75%, though, and CMS ran far too often:
-XX:CMSInitiatingOccupancyFraction=75
This made the "stop-the-world" pauses during CMS much shorter:
-XX:+CMSParallelRemarkEnabled
We use these in conjunction with CMS/ParNew and a 22gb heap (64gb total on the
box), with a 1.2G newSize/maxNewSize.
In case anyone else is having similar issues, we thought we would share our
experience with these newer options.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
