[ https://issues.apache.org/jira/browse/CASSANDRA-6127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Quentin Conner updated CASSANDRA-6127: -------------------------------------- Attachment: cpu-vs-token-graph.png The num_tokens setting has a mild impact on average cpu utilization. Please see the graph for the trend with N=256, 256 nodes. This graph does not characterize the bursty nature of any given node's CPU utilization. It does average the utilization over a 200 second period, taken at 10 second intervals using "sar -u". Since gossip heartbeat destinations are random, "unlucky" nodes will sometimes receive twice the gossip traffic and (10 second basis) CPU utilization has been casually observed @25% for N=256 and 60% for N=512. > vnodes don't scale to hundreds of nodes > --------------------------------------- > > Key: CASSANDRA-6127 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6127 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Any cluster that has vnodes and consists of hundreds of > physical nodes. > Reporter: Tupshin Harper > Assignee: Jonathan Ellis > Attachments: 2013-11-05_18-04-03_no_compression_cpu_time.png, > 2013-11-05_18-09-38_compression_on_cpu_time.png, 6000vnodes.patch, > AdjustableGossipPeriod.patch, cpu-vs-token-graph.png, > delayEstimatorUntilStatisticallyValid.patch, flaps-vs-tokens.png > > > There are a lot of gossip-related issues related to very wide clusters that > also have vnodes enabled. Let's use this ticket as a master in case there are > sub-tickets. > The most obvious symptom I've seen is with 1000 nodes in EC2 with m1.xlarge > instances. Each node configured with 32 vnodes. > Without vnodes, cluster spins up fine and is ready to handle requests within > 30 minutes or less. > With vnodes, nodes are reporting constant up/down flapping messages with no > external load on the cluster. After a couple of hours, they were still > flapping, had very high cpu load, and the cluster never looked like it was > going to stabilize or be useful for traffic. -- This message was sent by Atlassian JIRA (v6.1#6144)