[ 
https://issues.apache.org/jira/browse/CASSANDRA-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808328#comment-13808328
 ] 

Brandon Williams commented on CASSANDRA-6127:
---------------------------------------------

Let's move that discussion to CASSANDRA-4288, since that change is orthogonal 
to the actual problem we have here, regardless of whether it fixes it or just 
papers over the problem.  What we need to do next on this ticket is either 
correlate a thread dump to what is burning up CPU, or attach a debugger and see 
where the time is being spent.

> vnodes don't scale to hundreds of nodes
> ---------------------------------------
>
>                 Key: CASSANDRA-6127
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6127
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Any cluster that has vnodes and consists of hundreds of 
> physical nodes.
>            Reporter: Tupshin Harper
>            Assignee: Jonathan Ellis
>         Attachments: 6000vnodes.patch, AdjustableGossipPeriod.patch, 
> delayEstimatorUntilStatisticallyValid.patch
>
>
> There are a lot of gossip-related issues related to very wide clusters that 
> also have vnodes enabled. Let's use this ticket as a master in case there are 
> sub-tickets.
> The most obvious symptom I've seen is with 1000 nodes in EC2 with m1.xlarge 
> instances. Each node configured with 32 vnodes.
> Without vnodes, cluster spins up fine and is ready to handle requests within 
> 30 minutes or less. 
> With vnodes, nodes are reporting constant up/down flapping messages with no 
> external load on the cluster. After a couple of hours, they were still 
> flapping, had very high cpu load, and the cluster never looked like it was 
> going to stabilize or be useful for traffic.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Reply via email to