We're on the final stretch in getting our product database in Production with Solr. We have 13m "wide-ish" records with quite a few stored fields in a single index (no shards). We sort on at least a dozen fields and facet on 20-30. One thing that came up in QA testing is we were getting full gc's due to "promotion failed" conditions. This led us to believe we were dealing with large objects being created and a fragmented old generation. After improving, but not solving, the problem by tweaking "conventional" jvm parameters, our JVM expert learned about some newer tuning params included in Sun/Oracle's JDK 1.6.0_24 (we're running RHEL x64, but I think these are available on other platforms too):
These 3 options dramatically reduced the # objects getting promoted into the Old Gen, reducing fragmentation and CMS frequency & time: -XX:+UseStringCache -XX:+OptimizeStringConcat -XX:+UseCompressedStrings This uses compressed pointers on a 64-bit JVM, significantly reducing the memory & performance penalty in using a 64-bit jvm over 32-bit. This reduced our new GC (ParNew) time significantly: -XX:+UseCompressedOops The default for this was causing CMS to begin too late sometimes. (the documentated 68% proved false in our case. We figured it was defaulting close to 90%) Much lower than 75%, though, and CMS ran far too often: -XX:CMSInitiatingOccupancyFraction=75 This made the "stop-the-world" pauses during CMS much shorter: -XX:+CMSParallelRemarkEnabled We use these in conjunction with CMS/ParNew and a 22gb heap (64gb total on the box), with a 1.2G newSize/maxNewSize. In case anyone else is having similar issues, we thought we would share our experience with these newer options. James Dyer E-Commerce Systems Ingram Content Group (615) 213-4311
