Filippo Giunchedi has submitted this change and it was merged.
Change subject: Move Cassandra to g1gc collector and increase heap size
......................................................................
Move Cassandra to g1gc collector and increase heap size
The g1gc collector handles larger heaps better out of the box, and will be the
new default in Cassandra 3.0. We have tested it for a couple of hours in
production with the settings below, and it seems to have improved latencies
and reduced timeouts. It lets us increase the heap size slightly from 8g to
12g, which provides a bit of headroom for heavy compaction and write activity
on large instances.
Additionally, this patch slightly tweaks the number of concurrent writes
upwards from 32 to 48. This makes better use of the available SSD throughput,
and reduces the tendency to back up on the write stage.
Bug: T103161
Change-Id: I0ea1a46065f388cb42442b73d96ba239d3d07866
---
M hieradata/role/common/cassandra.yaml
M modules/cassandra/templates/cassandra-env.sh.erb
2 files changed, 19 insertions(+), 12 deletions(-)
Approvals:
Filippo Giunchedi: Verified; Looks good to me, approved
jenkins-bot: Verified
diff --git a/hieradata/role/common/cassandra.yaml
b/hieradata/role/common/cassandra.yaml
index 21609dd..72bdb1d 100644
--- a/hieradata/role/common/cassandra.yaml
+++ b/hieradata/role/common/cassandra.yaml
@@ -11,11 +11,12 @@
- restbase1004.eqiad.wmnet
- restbase1005.eqiad.wmnet
- restbase1006.eqiad.wmnet
-cassandra::max_heap_size: 8g
+cassandra::max_heap_size: 12g
# 1/4 heap size, no more than 100m/thread
cassandra::heap_newsize: 2048m
cassandra::compaction_throughput_mb_per_sec: 160
cassandra::concurrent_compactors: 10
+cassandra::concurrent_writes: 48
cassandra::dc: "%{::site}"
cassandra::cluster_name: "%{::site}"
diff --git a/modules/cassandra/templates/cassandra-env.sh.erb
b/modules/cassandra/templates/cassandra-env.sh.erb
index d78659d..d1720fa 100644
--- a/modules/cassandra/templates/cassandra-env.sh.erb
+++ b/modules/cassandra/templates/cassandra-env.sh.erb
@@ -196,7 +196,7 @@
# out.
JVM_OPTS="$JVM_OPTS -Xms${MAX_HEAP_SIZE}"
JVM_OPTS="$JVM_OPTS -Xmx${MAX_HEAP_SIZE}"
-JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
+#JVM_OPTS="$JVM_OPTS -Xmn${HEAP_NEWSIZE}"
JVM_OPTS="$JVM_OPTS -XX:+HeapDumpOnOutOfMemoryError"
# set jvm HeapDumpPath with CASSANDRA_HEAPDUMP_DIR
@@ -214,16 +214,22 @@
JVM_OPTS="$JVM_OPTS -XX:StringTableSize=1000003"
# GC tuning options
-JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
-JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
-JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
-JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
-JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=5"
-JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=78"
-JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
-JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
-JVM_OPTS="$JVM_OPTS -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler"
-JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=10000"
+#JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
+#JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
+#JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
+#JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
+#JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=5"
+#JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=78"
+#JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
+#JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
+#JVM_OPTS="$JVM_OPTS -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler"
+#JVM_OPTS="$JVM_OPTS -XX:CMSWaitDuration=10000"
+
+# GC tuning options for g1gc
+# See https://phabricator.wikimedia.org/T103161
+JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
+JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=5"
+JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=300"
# note: bash evals '1.7.x' as > '1.7' so this is really a >= 1.7 jvm check
if [ "$JVM_ARCH" = "64-Bit" ] ; then
--
To view, visit https://gerrit.wikimedia.org/r/221993
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I0ea1a46065f388cb42442b73d96ba239d3d07866
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: GWicke <gwi...@wikimedia.org>
Gerrit-Reviewer: Eevans <eev...@wikimedia.org>
Gerrit-Reviewer: Filippo Giunchedi <fgiunch...@wikimedia.org>
Gerrit-Reviewer: GWicke <gwi...@wikimedia.org>
Gerrit-Reviewer: Mobrovac <mobro...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
