[jira] [Commented] (CASSANDRA-5272) Hinted Handoff Throttle based on cluster size
[ https://issues.apache.org/jira/browse/CASSANDRA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669639#comment-13669639 ] Jonathan Ellis commented on CASSANDRA-5272: --- I dunno, then you have to worry about flooding again with edge cases like CASSANDRA-3533. I'll go ahead and commit this and we can try to be more sophisticated later if warranted. > Hinted Handoff Throttle based on cluster size > - > > Key: CASSANDRA-5272 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5272 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.2.0 >Reporter: Rick Branson >Assignee: Jonathan Ellis >Priority: Minor > Labels: lhf > Fix For: 1.2.6 > > Attachments: 5272.txt > > > For a 12-node EC2 m1.xlarge cluster, restarting a node causes it to get > completely overloaded with the default 2-thread, 1024KB setting in 1.2.x. > This seemed to be a smaller problem when it was 6-nodes, but still required > us to abort handoffs. The old defaults in 1.1.x were WAY more conservative. > I've dropped this way down to 128KB on our production cluster which is really > conservative, but appears to have solved it. The default seems way too high > on any cluster that is non-trivial in size. > After putting some thought to this, it seems that this should really be based > on cluster size, making the throttle a "target" for how much write load a > single node can swallow. As the cluster grows, the amount of hints that can > be delivered by each other node in the cluster goes down, so the throttle > should self-adjust to take that into account. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5272) Hinted Handoff Throttle based on cluster size
[ https://issues.apache.org/jira/browse/CASSANDRA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668609#comment-13668609 ] Brandon Williams commented on CASSANDRA-5272: - Would it be better perhaps to throttle by the amount of live nodes? It won't always be accurate, but if you lost a WAN link between DCs you'd be throttling a lot more than needed at the time. > Hinted Handoff Throttle based on cluster size > - > > Key: CASSANDRA-5272 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5272 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.2.0 >Reporter: Rick Branson >Assignee: Jonathan Ellis >Priority: Minor > Labels: lhf > Fix For: 1.2.6 > > Attachments: 5272.txt > > > For a 12-node EC2 m1.xlarge cluster, restarting a node causes it to get > completely overloaded with the default 2-thread, 1024KB setting in 1.2.x. > This seemed to be a smaller problem when it was 6-nodes, but still required > us to abort handoffs. The old defaults in 1.1.x were WAY more conservative. > I've dropped this way down to 128KB on our production cluster which is really > conservative, but appears to have solved it. The default seems way too high > on any cluster that is non-trivial in size. > After putting some thought to this, it seems that this should really be based > on cluster size, making the throttle a "target" for how much write load a > single node can swallow. As the cluster grows, the amount of hints that can > be delivered by each other node in the cluster goes down, so the throttle > should self-adjust to take that into account. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5272) Hinted Handoff Throttle based on cluster size
[ https://issues.apache.org/jira/browse/CASSANDRA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13668584#comment-13668584 ] Rick Branson commented on CASSANDRA-5272: - There's a small spelling error "eeach" in the cassandra.yaml. Also, should we add a line to NEWS to describe this behavior since it's going to change the setting in place? (i.e. we will need to bump our throttle up a bunch when rolling this out). > Hinted Handoff Throttle based on cluster size > - > > Key: CASSANDRA-5272 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5272 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.2.0 >Reporter: Rick Branson >Assignee: Jonathan Ellis >Priority: Minor > Labels: lhf > Fix For: 1.2.6 > > Attachments: 5272.txt > > > For a 12-node EC2 m1.xlarge cluster, restarting a node causes it to get > completely overloaded with the default 2-thread, 1024KB setting in 1.2.x. > This seemed to be a smaller problem when it was 6-nodes, but still required > us to abort handoffs. The old defaults in 1.1.x were WAY more conservative. > I've dropped this way down to 128KB on our production cluster which is really > conservative, but appears to have solved it. The default seems way too high > on any cluster that is non-trivial in size. > After putting some thought to this, it seems that this should really be based > on cluster size, making the throttle a "target" for how much write load a > single node can swallow. As the cluster grows, the amount of hints that can > be delivered by each other node in the cluster goes down, so the throttle > should self-adjust to take that into account. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5272) Hinted Handoff Throttle based on cluster size
[ https://issues.apache.org/jira/browse/CASSANDRA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13664482#comment-13664482 ] Jonathan Ellis commented on CASSANDRA-5272: --- Correction; patch is against trunk but should be trivial to retrofit if necessary. > Hinted Handoff Throttle based on cluster size > - > > Key: CASSANDRA-5272 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5272 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.2.0 >Reporter: Rick Branson >Assignee: Jonathan Ellis >Priority: Minor > Labels: lhf > Fix For: 1.2.6 > > Attachments: 5272.txt > > > For a 12-node EC2 m1.xlarge cluster, restarting a node causes it to get > completely overloaded with the default 2-thread, 1024KB setting in 1.2.x. > This seemed to be a smaller problem when it was 6-nodes, but still required > us to abort handoffs. The old defaults in 1.1.x were WAY more conservative. > I've dropped this way down to 128KB on our production cluster which is really > conservative, but appears to have solved it. The default seems way too high > on any cluster that is non-trivial in size. > After putting some thought to this, it seems that this should really be based > on cluster size, making the throttle a "target" for how much write load a > single node can swallow. As the cluster grows, the amount of hints that can > be delivered by each other node in the cluster goes down, so the throttle > should self-adjust to take that into account. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
