Re: Crazy Solr startup params, please review

2014-11-05 Thread Jan Høydahl
Can we in a smart way build these best practices into the script, by testing 
JVM version and (un)setting some of these automatically based on version? The 
solr.in.* config could have a new option GC_TUNE_AUTO="true" and GC_TUNE 
commented out. Then if auto is enabled, the bin/solr scripts will decide what 
flags to set. Experts can override. Also related is SOLR-6705, some options 
should be set automatically only for some JVM versions.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 5. nov. 2014 kl. 16.53 skrev Uwe Schindler :
> 
> Hi,
> 
> I just reviewed the solr.sh.in and solr.cmd.in scripts in solr/bin folder. 
> The following was committed but never discussed:
> 
> # These GC settings have shown to work well for a number of common Solr 
> workloads
> GC_TUNE="-XX:-UseSuperWord \
> -XX:NewRatio=3 \
> -XX:SurvivorRatio=4 \
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
> -XX:+UseConcMarkSweepGC \
> -XX:+UseParNewGC \
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> -XX:+CMSScavengeBeforeRemark \
> -XX:PretenureSizeThreshold=64m \
> -XX:CMSFullGCsBeforeCompaction=1 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=50 \
> -XX:CMSTriggerPermRatio=80 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled \
> -XX:+ParallelRefProcEnabled \
> -XX:+AggressiveOpts"
> 
> This is horrible, because of our experience with Hotspot bugs:
> 
> -XX:+AggressiveOpts
> This option is vry risky and speed improvements are marginal. PLEASE 
> DON'T DO THIS. If people want the new features they should wait for later 
> Java releases and the new features are tested. See several tasks about the 
> Java 7 disaster! In fact we had reports at Java 6 times when people had 
> enabled this and were affected by the Java 7 GA bugs already in Java 6 and 
> corrupted their indexes!!!
> 
> -XX:-UseSuperWord
> If you have a Haswell CPU, all other improvements in these command line 
> settings are eaten up by this flag! If you have 7u55 at minimum, you should 
> never disable this. Things like BooleanFilter and other bitset operations are 
> up to 2 times faster with Java 7u55 on Haswell CPUs and later!!! This setting 
> only makes sense if you have one of those buggy JDKs (7u40 to 7u51). In all 
> other cases this slows down enormous!
> In addition, enabling this option may break JDKs before 7u40 (this option was 
> added in 7u40), so breaks:
>> Unrecognized VM option 'UseSuperWord'
> 
> It would be good, if we could fix the startup scripts not not have options, 
> which may also break with JDK 8 or later!
> 
> Uwe
> 
> -
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: u...@thetaphi.de
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Crazy Solr startup params, please review

2014-11-05 Thread Uwe Schindler
Hi,

I just reviewed the solr.sh.in and solr.cmd.in scripts in solr/bin folder. The 
following was committed but never discussed:

# These GC settings have shown to work well for a number of common Solr 
workloads
GC_TUNE="-XX:-UseSuperWord \
-XX:NewRatio=3 \
-XX:SurvivorRatio=4 \
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:+CMSScavengeBeforeRemark \
-XX:PretenureSizeThreshold=64m \
-XX:CMSFullGCsBeforeCompaction=1 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=50 \
-XX:CMSTriggerPermRatio=80 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled \
-XX:+ParallelRefProcEnabled \
-XX:+AggressiveOpts"

This is horrible, because of our experience with Hotspot bugs:

-XX:+AggressiveOpts
This option is vry risky and speed improvements are marginal. PLEASE DON'T 
DO THIS. If people want the new features they should wait for later Java 
releases and the new features are tested. See several tasks about the Java 7 
disaster! In fact we had reports at Java 6 times when people had enabled this 
and were affected by the Java 7 GA bugs already in Java 6 and corrupted their 
indexes!!!

-XX:-UseSuperWord
If you have a Haswell CPU, all other improvements in these command line 
settings are eaten up by this flag! If you have 7u55 at minimum, you should 
never disable this. Things like BooleanFilter and other bitset operations are 
up to 2 times faster with Java 7u55 on Haswell CPUs and later!!! This setting 
only makes sense if you have one of those buggy JDKs (7u40 to 7u51). In all 
other cases this slows down enormous!
In addition, enabling this option may break JDKs before 7u40 (this option was 
added in 7u40), so breaks:
> Unrecognized VM option 'UseSuperWord'

It would be good, if we could fix the startup scripts not not have options, 
which may also break with JDK 8 or later!

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de




-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org