[
https://issues.apache.org/jira/browse/CASSANDRA-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009705#comment-15009705
]
Tyler Hobbs commented on CASSANDRA-6992:
----------------------------------------
bq. We could also provide two tuning properties
I feel like adding config properties may just add unnecessary complexity to the
config for something that would rarely need to be tuned, especially if our
defaults are conservative. So, my personal preference is to pick conservative
defaults and wait to see if anybody really needs to tune this.
> Bootstrap on vnodes clusters can cause stampeding/storm behavior
> ----------------------------------------------------------------
>
> Key: CASSANDRA-6992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6992
> Project: Cassandra
> Issue Type: Improvement
> Components: Coordination, Streaming and Messaging
> Environment: Various vnodes-enabled clusters in EC2, m1.xlarge and
> hi1.4xlarge, ~3000-8000 tokens.
> Reporter: Rick Branson
> Assignee: Paulo Motta
> Priority: Minor
>
> Assuming this is an issue with vnodes clusters because
> SSTableReader#getPositionsForRanges is more expensive to compute with 256x
> the ranges, but could be wrong. On even well-provisioned hosts, this can
> cause a severe spike in network throughput & CPU utilization from a storm of
> flushes, which impacts long-tail times pretty badly. On weaker hosts (like
> m1.xlarge with ~500GB of data), it can result in minutes of churn while the
> node gets through StreamOut#createPendingFiles. This *might* be better in
> 2.0, but it's probably still reproducible because the bootstrapping node
> sends out all of it's streaming requests at once.
> I'm thinking that this could be staggered at the bootstrapping node to avoid
> the simultaneous spike across the whole cluster. Not sure on how to stagger
> it besides something very naive like one-at-a-time with a pause. Maybe this
> should also be throttled in StreamOut#createPendingFiles on the out-streaming
> host? Any thoughts?
> From the stack dump of one of our weaker nodes that was struggling for a few
> minutes just starting the StreamOut:
> "MiscStage:1" daemon prio=10 tid=0x000000000292f000 nid=0x688 runnable
> [0x00007f7b03df6000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.cassandra.utils.ByteBufferUtil.readShortLength(ByteBufferUtil.java:361)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
> at
> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:187)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:125)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:889)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:790)
> at
> org.apache.cassandra.io.sstable.SSTableReader.getPositionsForRanges(SSTableReader.java:730)
> at
> org.apache.cassandra.streaming.StreamOut.createPendingFiles(StreamOut.java:172)
> at
> org.apache.cassandra.streaming.StreamOut.transferSSTables(StreamOut.java:157)
> at
> org.apache.cassandra.streaming.StreamOut.transferRanges(StreamOut.java:148)
> at
> org.apache.cassandra.streaming.StreamOut.transferRanges(StreamOut.java:116)
> at
> org.apache.cassandra.streaming.StreamRequestVerbHandler.doVerb(StreamRequestVerbHandler.java:44)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
