GWicke has uploaded a new change for review. https://gerrit.wikimedia.org/r/223722
Change subject: Reduce the compaction throughput limit to 80mb/s ...................................................................... Reduce the compaction throughput limit to 80mb/s With our current nodes iowait occasionally hits double digits when Cassandra is reading more than 80mb/s. This means that empirically 80mb/s seems to be a good compaction throughput limit that leaves sufficient headroom for request activity and doesn't overwhelm the GC, but also makes sufficient compaction progress. This number has been tested in production by setting the limit dynamically on all nodes: for i in `seq 1 6`; do nodetool -h restbase100$i.eqiad.wmnet setcompactionthroughput 80 done Dynamic changes to the compaction throughput only seem to affect newly started compactions, so allow for some time for all ongoing compactions (see `nodetool compactionstats`) to finish before judging the setting's effect. Change-Id: Iaf55ac89fe01dbb634b755b71144ab5063857af8 --- M hieradata/role/common/cassandra.yaml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/22/223722/1 diff --git a/hieradata/role/common/cassandra.yaml b/hieradata/role/common/cassandra.yaml index 86b949e..4b34133 100644 --- a/hieradata/role/common/cassandra.yaml +++ b/hieradata/role/common/cassandra.yaml @@ -14,7 +14,7 @@ cassandra::max_heap_size: 16g # 1/4 heap size, no more than 100m/thread cassandra::heap_newsize: 2048m -cassandra::compaction_throughput_mb_per_sec: 160 +cassandra::compaction_throughput_mb_per_sec: 80 cassandra::concurrent_compactors: 10 cassandra::concurrent_writes: 128 cassandra::concurrent_reads: 96 -- To view, visit https://gerrit.wikimedia.org/r/223722 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Iaf55ac89fe01dbb634b755b71144ab5063857af8 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: GWicke <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
