Re: Informal poll on running Solr 4 on Java 7 with G1GC
And here is our most recent experience with G1, although not with Solr, but with HBase: http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Fri, Jun 21, 2013 at 11:33 AM, Walter Underwood wun...@wunderwood.org wrote: On 6/20/2013 10:22 PM, William Bell wrote: It would be good to see some CMS configs too... Can you send your java params? Here is what we use in production. We run multiple collections with small documents. One is 3M docs, one is 9M, one is 2M, and the other three are small. We use Amazon m1.xlarge instances (4CPU, 15GB). These options were developed with our load test. That is based on a full day of queries. We use JMeter to send queries at a constant rate that takes the CPU to between 50% and 75% busy. We measure 95th and 99th percentiles for response time. We enable ExplicitGCInvokesConcurrent because some monitoring software was calling System.gc() to get accurate memory numbers. That was causing notable pauses in service and messing up our 99th percentile. Alternatively, you could disable those entirely with the flag DisableExplicitGC. The new size is large so that all the allocations needed to handle a single request can fit in new space. We really do not want per-request data being allocated in tenured space. New needs to be big enough to handle multiple simultaneous requests. export CATALINA_OPTS=$CATALINA_OPTS -d64 export CATALINA_OPTS=$CATALINA_OPTS -server export CATALINA_OPTS=$CATALINA_OPTS -Xms8g export CATALINA_OPTS=$CATALINA_OPTS -Xmx8g export CATALINA_OPTS=$CATALINA_OPTS -XX:NewSize=2048m export CATALINA_OPTS=$CATALINA_OPTS -XX:MaxPermSize=256m export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseConcMarkSweepGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseParNewGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+ExplicitGCInvokesConcurrent export CATALINA_OPTS=$CATALINA_OPTS -verbose:gc export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCDetails export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCTimeStamps export CATALINA_OPTS=$CATALINA_OPTS -XX:-TraceClassUnloading export CATALINA_OPTS=$CATALINA_OPTS -Xloggc:$CATALINA_HOME/logs/gc.log export CATALINA_OPTS=$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError export CATALINA_OPTS=$CATALINA_OPTS -XX:HeapDumpPath=$CATALINA_HOME/logs/ We used to include these options, but they default to enabled in Java 1.7 Update 17. -XX:+DoEscapeAnalysis -XX:+CMSParallelRemarkEnabled -XX:+UseCompressedOops wunder
Re: Informal poll on running Solr 4 on Java 7 with G1GC
Interesting. It seems to spend more time in GC, but the major GCs aren't any faster. They are more consistent. I notice that SPM shows average collection time. This is not a particularly useful number. It should use median and percentiles. For a one-sided distribution, never use mean (average), always use median and percentiles. This is one of my basic tests for monitoring products. If they show averages for response time (or other durations), they are doing it wrong and need to learn more about statistics. wunder On Jun 24, 2013, at 8:59 AM, Otis Gospodnetic wrote: And here is our most recent experience with G1, although not with Solr, but with HBase: http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Fri, Jun 21, 2013 at 11:33 AM, Walter Underwood wun...@wunderwood.org wrote: On 6/20/2013 10:22 PM, William Bell wrote: It would be good to see some CMS configs too... Can you send your java params? Here is what we use in production. We run multiple collections with small documents. One is 3M docs, one is 9M, one is 2M, and the other three are small. We use Amazon m1.xlarge instances (4CPU, 15GB). These options were developed with our load test. That is based on a full day of queries. We use JMeter to send queries at a constant rate that takes the CPU to between 50% and 75% busy. We measure 95th and 99th percentiles for response time. We enable ExplicitGCInvokesConcurrent because some monitoring software was calling System.gc() to get accurate memory numbers. That was causing notable pauses in service and messing up our 99th percentile. Alternatively, you could disable those entirely with the flag DisableExplicitGC. The new size is large so that all the allocations needed to handle a single request can fit in new space. We really do not want per-request data being allocated in tenured space. New needs to be big enough to handle multiple simultaneous requests. export CATALINA_OPTS=$CATALINA_OPTS -d64 export CATALINA_OPTS=$CATALINA_OPTS -server export CATALINA_OPTS=$CATALINA_OPTS -Xms8g export CATALINA_OPTS=$CATALINA_OPTS -Xmx8g export CATALINA_OPTS=$CATALINA_OPTS -XX:NewSize=2048m export CATALINA_OPTS=$CATALINA_OPTS -XX:MaxPermSize=256m export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseConcMarkSweepGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseParNewGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+ExplicitGCInvokesConcurrent export CATALINA_OPTS=$CATALINA_OPTS -verbose:gc export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCDetails export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCTimeStamps export CATALINA_OPTS=$CATALINA_OPTS -XX:-TraceClassUnloading export CATALINA_OPTS=$CATALINA_OPTS -Xloggc:$CATALINA_HOME/logs/gc.log export CATALINA_OPTS=$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError export CATALINA_OPTS=$CATALINA_OPTS -XX:HeapDumpPath=$CATALINA_HOME/logs/ We used to include these options, but they default to enabled in Java 1.7 Update 17. -XX:+DoEscapeAnalysis -XX:+CMSParallelRemarkEnabled -XX:+UseCompressedOops wunder -- Walter Underwood wun...@wunderwood.org
Re: Informal poll on running Solr 4 on Java 7 with G1GC
Yup, known stuff, on TODO. Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Mon, Jun 24, 2013 at 12:50 PM, Walter Underwood wun...@wunderwood.org wrote: Interesting. It seems to spend more time in GC, but the major GCs aren't any faster. They are more consistent. I notice that SPM shows average collection time. This is not a particularly useful number. It should use median and percentiles. For a one-sided distribution, never use mean (average), always use median and percentiles. This is one of my basic tests for monitoring products. If they show averages for response time (or other durations), they are doing it wrong and need to learn more about statistics. wunder On Jun 24, 2013, at 8:59 AM, Otis Gospodnetic wrote: And here is our most recent experience with G1, although not with Solr, but with HBase: http://blog.sematext.com/2013/06/24/g1-cms-java-garbage-collector/ Otis -- Solr ElasticSearch Support -- http://sematext.com/ Performance Monitoring -- http://sematext.com/spm On Fri, Jun 21, 2013 at 11:33 AM, Walter Underwood wun...@wunderwood.org wrote: On 6/20/2013 10:22 PM, William Bell wrote: It would be good to see some CMS configs too... Can you send your java params? Here is what we use in production. We run multiple collections with small documents. One is 3M docs, one is 9M, one is 2M, and the other three are small. We use Amazon m1.xlarge instances (4CPU, 15GB). These options were developed with our load test. That is based on a full day of queries. We use JMeter to send queries at a constant rate that takes the CPU to between 50% and 75% busy. We measure 95th and 99th percentiles for response time. We enable ExplicitGCInvokesConcurrent because some monitoring software was calling System.gc() to get accurate memory numbers. That was causing notable pauses in service and messing up our 99th percentile. Alternatively, you could disable those entirely with the flag DisableExplicitGC. The new size is large so that all the allocations needed to handle a single request can fit in new space. We really do not want per-request data being allocated in tenured space. New needs to be big enough to handle multiple simultaneous requests. export CATALINA_OPTS=$CATALINA_OPTS -d64 export CATALINA_OPTS=$CATALINA_OPTS -server export CATALINA_OPTS=$CATALINA_OPTS -Xms8g export CATALINA_OPTS=$CATALINA_OPTS -Xmx8g export CATALINA_OPTS=$CATALINA_OPTS -XX:NewSize=2048m export CATALINA_OPTS=$CATALINA_OPTS -XX:MaxPermSize=256m export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseConcMarkSweepGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseParNewGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+ExplicitGCInvokesConcurrent export CATALINA_OPTS=$CATALINA_OPTS -verbose:gc export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCDetails export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCTimeStamps export CATALINA_OPTS=$CATALINA_OPTS -XX:-TraceClassUnloading export CATALINA_OPTS=$CATALINA_OPTS -Xloggc:$CATALINA_HOME/logs/gc.log export CATALINA_OPTS=$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError export CATALINA_OPTS=$CATALINA_OPTS -XX:HeapDumpPath=$CATALINA_HOME/logs/ We used to include these options, but they default to enabled in Java 1.7 Update 17. -XX:+DoEscapeAnalysis -XX:+CMSParallelRemarkEnabled -XX:+UseCompressedOops wunder -- Walter Underwood wun...@wunderwood.org
Re: Informal poll on running Solr 4 on Java 7 with G1GC
Hi Shawn, some good information about G1 tuning, may be something for the wiki about GC tuning. Am 20.06.2013 18:01, schrieb Shawn Heisey: On 6/20/2013 8:02 AM, John Nielsen wrote: ... ... When you take a look at the overall stats and the memory graph over time, G1 looks way better. Unfortunately GC with any collector does sometimes get bad, and when that happens, un-tuned G1 is a little worse than un-tuned CMS. Perhaps if G1 were tuned, it would be really good, but I haven't been able to find any information on how to tune G1. A very good blog about GC tuning and CMS versa G1 http://blog.mgm-tp.com/2013/03/garbage-collection-tuning/ Everything about G1 tuning http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/G1GettingStarted/index.html And if you are good in German language http://www.angelikalanger.com/Articles/EffectiveJava/55.GC.G1.Overview/55.GC.G1.Overview.html Regards Bernd
Re: Informal poll on running Solr 4 on Java 7 with G1GC
On 6/20/2013 10:22 PM, William Bell wrote: It would be good to see some CMS configs too... Can you send your java params? I've got it documented on my personal wiki page: https://wiki.apache.org/solr/ShawnHeisey#GC_Tuning Thanks, Shawn
Re: Informal poll on running Solr 4 on Java 7 with G1GC
On 6/20/2013 10:22 PM, William Bell wrote: It would be good to see some CMS configs too... Can you send your java params? Here is what we use in production. We run multiple collections with small documents. One is 3M docs, one is 9M, one is 2M, and the other three are small. We use Amazon m1.xlarge instances (4CPU, 15GB). These options were developed with our load test. That is based on a full day of queries. We use JMeter to send queries at a constant rate that takes the CPU to between 50% and 75% busy. We measure 95th and 99th percentiles for response time. We enable ExplicitGCInvokesConcurrent because some monitoring software was calling System.gc() to get accurate memory numbers. That was causing notable pauses in service and messing up our 99th percentile. Alternatively, you could disable those entirely with the flag DisableExplicitGC. The new size is large so that all the allocations needed to handle a single request can fit in new space. We really do not want per-request data being allocated in tenured space. New needs to be big enough to handle multiple simultaneous requests. export CATALINA_OPTS=$CATALINA_OPTS -d64 export CATALINA_OPTS=$CATALINA_OPTS -server export CATALINA_OPTS=$CATALINA_OPTS -Xms8g export CATALINA_OPTS=$CATALINA_OPTS -Xmx8g export CATALINA_OPTS=$CATALINA_OPTS -XX:NewSize=2048m export CATALINA_OPTS=$CATALINA_OPTS -XX:MaxPermSize=256m export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseConcMarkSweepGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+UseParNewGC export CATALINA_OPTS=$CATALINA_OPTS -XX:+ExplicitGCInvokesConcurrent export CATALINA_OPTS=$CATALINA_OPTS -verbose:gc export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCDetails export CATALINA_OPTS=$CATALINA_OPTS -XX:+PrintGCTimeStamps export CATALINA_OPTS=$CATALINA_OPTS -XX:-TraceClassUnloading export CATALINA_OPTS=$CATALINA_OPTS -Xloggc:$CATALINA_HOME/logs/gc.log export CATALINA_OPTS=$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError export CATALINA_OPTS=$CATALINA_OPTS -XX:HeapDumpPath=$CATALINA_HOME/logs/ We used to include these options, but they default to enabled in Java 1.7 Update 17. -XX:+DoEscapeAnalysis -XX:+CMSParallelRemarkEnabled -XX:+UseCompressedOops wunder
Re: Informal poll on running Solr 4 on Java 7 with G1GC
Am 20.06.2013 00:18, schrieb Timothy Potter: I'm sure there's some site to do this but wanted to get a feel for who's running Solr 4 on Java 7 with G1 gc enabled? Cheers, Tim Currently using Solr 4.2.1 in production with Oracle Java(TM) SE Runtime Environment (build 1.7.0_07-b10) and using G1GC without any options. Linux 2.6.32.23-0.3-xen SMP x86_64 GNU/Linux. Performs better than CMS (with several tuning options). Very little sawtooth, smaller faster GCs. 1 Master/3 Slave System, 128.97 GB index, 46.3 mio. docs. Bernd
Re: Informal poll on running Solr 4 on Java 7 with G1GC
We used to use G1, but recently went back to CMS. G1 gave us too long stop-the-world events. CMS uses more ressources for the same work, but it is more predictable and we get better worst-case performance out of it. Med venlig hilsen / Best regards John Nielsen Programmer MCB A/S Enghaven 15 DK-7500 Holstebro Kundeservice: +45 9610 2824 p...@mcb.dk www.mcb.dk On 20-06-2013 00:18, Timothy Potter wrote: I'm sure there's some site to do this but wanted to get a feel for who's running Solr 4 on Java 7 with G1 gc enabled? Cheers, Tim
Re: Informal poll on running Solr 4 on Java 7 with G1GC
On 6/20/2013 8:02 AM, John Nielsen wrote: We used to use G1, but recently went back to CMS. G1 gave us too long stop-the-world events. CMS uses more ressources for the same work, but it is more predictable and we get better worst-case performance out of it. This is exactly the behavior I saw. When you take a look at the overall stats and the memory graph over time, G1 looks way better. Unfortunately GC with any collector does sometimes get bad, and when that happens, un-tuned G1 is a little worse than un-tuned CMS. Perhaps if G1 were tuned, it would be really good, but I haven't been able to find any information on how to tune G1. jHiccup or gclogviewer can give you really good insight into how your GC is doing in both average and worst-case scenarios. jHiccup is a wrapper for your program and gclogviewer draws graphs from GC logs. I'm not sure whether gclogviewer works with G1 logs or not, but I know that jHiccup will work with G1. http://www.azulsystems.com/downloads/jHiccup http://code.google.com/p/gclogviewer/downloads/list http://code.google.com/p/gclogviewer/source/checkout http://code.google.com/p/gclogviewer/issues/detail?id=7 Thanks, Shawn
Re: Informal poll on running Solr 4 on Java 7 with G1GC
Awesome info, thanks Shawn! I'll post back my results with G1 after we've had some time to analyze it in production. On Thu, Jun 20, 2013 at 11:01 AM, Shawn Heisey s...@elyograg.org wrote: On 6/20/2013 8:02 AM, John Nielsen wrote: We used to use G1, but recently went back to CMS. G1 gave us too long stop-the-world events. CMS uses more ressources for the same work, but it is more predictable and we get better worst-case performance out of it. This is exactly the behavior I saw. When you take a look at the overall stats and the memory graph over time, G1 looks way better. Unfortunately GC with any collector does sometimes get bad, and when that happens, un-tuned G1 is a little worse than un-tuned CMS. Perhaps if G1 were tuned, it would be really good, but I haven't been able to find any information on how to tune G1. jHiccup or gclogviewer can give you really good insight into how your GC is doing in both average and worst-case scenarios. jHiccup is a wrapper for your program and gclogviewer draws graphs from GC logs. I'm not sure whether gclogviewer works with G1 logs or not, but I know that jHiccup will work with G1. http://www.azulsystems.com/downloads/jHiccup http://code.google.com/p/gclogviewer/downloads/list http://code.google.com/p/gclogviewer/source/checkout http://code.google.com/p/gclogviewer/issues/detail?id=7 Thanks, Shawn
RE: Informal poll on running Solr 4 on Java 7 with G1GC
I've been trying it out on solr 3.6.1 with a 32GB heap and G1GC seems to be more prone to OOMEs than CMS. I have been running it on one slave box in our farm and the rest of the slaves are still on CMS and three times now it has gone OOM on me whereas the rest of our slaves kept chugging along with no errors. I even went from no other tuning params to using these suggested on Shawns wiki page here and that didn't help either, still got some OOMs. I'm giving it a 'fail' pretty soon here. -XX:+AggressiveOpts -XX:+HeapDumpOnOutOfMemoryError -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+UseG1GC -XX:+UseStringCache -XX:-UseSplitVerifier -XX:MaxGCPauseMillis=50 http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning Thanks Robi -Original Message- From: Timothy Potter [mailto:thelabd...@gmail.com] Sent: Thursday, June 20, 2013 9:22 AM To: solr-user@lucene.apache.org Subject: Re: Informal poll on running Solr 4 on Java 7 with G1GC Awesome info, thanks Shawn! I'll post back my results with G1 after we've had some time to analyze it in production. On Thu, Jun 20, 2013 at 11:01 AM, Shawn Heisey s...@elyograg.org wrote: On 6/20/2013 8:02 AM, John Nielsen wrote: We used to use G1, but recently went back to CMS. G1 gave us too long stop-the-world events. CMS uses more ressources for the same work, but it is more predictable and we get better worst-case performance out of it. This is exactly the behavior I saw. When you take a look at the overall stats and the memory graph over time, G1 looks way better. Unfortunately GC with any collector does sometimes get bad, and when that happens, un-tuned G1 is a little worse than un-tuned CMS. Perhaps if G1 were tuned, it would be really good, but I haven't been able to find any information on how to tune G1. jHiccup or gclogviewer can give you really good insight into how your GC is doing in both average and worst-case scenarios. jHiccup is a wrapper for your program and gclogviewer draws graphs from GC logs. I'm not sure whether gclogviewer works with G1 logs or not, but I know that jHiccup will work with G1. http://www.azulsystems.com/downloads/jHiccup http://code.google.com/p/gclogviewer/downloads/list http://code.google.com/p/gclogviewer/source/checkout http://code.google.com/p/gclogviewer/issues/detail?id=7 Thanks, Shawn
Re: Informal poll on running Solr 4 on Java 7 with G1GC
It would be good to see some CMS configs too... Can you send your java params? On Wed, Jun 19, 2013 at 8:46 PM, Shawn Heisey s...@elyograg.org wrote: On 6/19/2013 4:18 PM, Timothy Potter wrote: I'm sure there's some site to do this but wanted to get a feel for who's running Solr 4 on Java 7 with G1 gc enabled? I have tried it, but found that G1 didn't give me any better GC pause characteristics than CMS without tuning, and may have actually been worse. Now I use CMS with several tuning options. Thanks, Shawn -- Bill Bell billnb...@gmail.com cell 720-256-8076
Informal poll on running Solr 4 on Java 7 with G1GC
I'm sure there's some site to do this but wanted to get a feel for who's running Solr 4 on Java 7 with G1 gc enabled? Cheers, Tim
Re: Informal poll on running Solr 4 on Java 7 with G1GC
On 6/19/2013 4:18 PM, Timothy Potter wrote: I'm sure there's some site to do this but wanted to get a feel for who's running Solr 4 on Java 7 with G1 gc enabled? I have tried it, but found that G1 didn't give me any better GC pause characteristics than CMS without tuning, and may have actually been worse. Now I use CMS with several tuning options. Thanks, Shawn
