[
https://issues.apache.org/jira/browse/CASSANDRA-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15064029#comment-15064029
]
Branimir Lambov commented on CASSANDRA-9258:
--------------------------------------------
Thank you for writing the benchmark. There are a couple problems with it,
though:
- the generated ranges cover the space up to {{maxValue * 10 + 15}}, but you
only test tokens between 0 and {{maxValue}} which is skewing the result;
- ranges are non-overlapping except for a lot of wrap-around ones -- to better
simulate the target usage, I'd just use a single wrap-around range;
- you could use more than one address for some of the ranges;
- make sure it follows the code conventions;
- could you post some before and after results?
A small correction on one of the changes you made on my request:
{{addIntersection}} was meant to just do the {{for}} loop, so that it could be
called like this:
{code}
if (ascendingTailMap.size() < descendingTailMap.size())
addIntersection(endpoints, ascendingTailMap, descendingTailMap);
else
addIntersection(endpoints, descendingTailMap, ascendingTailMap);
{code}
> Range movement causes CPU & performance impact
> ----------------------------------------------
>
> Key: CASSANDRA-9258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9258
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.1.4
> Reporter: Rick Branson
> Assignee: Dikang Gu
> Fix For: 2.1.x
>
> Attachments: 0001-pending-ranges-map.patch,
> 0001-pending-ranges-maps-for-2.2.patch, Screenshot 2015-12-16 16.11.36.png,
> Screenshot 2015-12-16 16.11.51.png
>
>
> Observing big CPU & latency regressions when doing range movements on
> clusters with many tens of thousands of vnodes. See CPU usage increase by
> ~80% when a single node is being replaced.
> Top methods are:
> 1) Ljava/math/BigInteger;.compareTo in
> Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo
> 2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in
> Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
> 3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in
> Lorg/apache/cassandra/dht/Range;.contains
> Here's a sample stack from a thread dump:
> {code}
> "Thrift:50673" daemon prio=10 tid=0x00007f2f20164800 nid=0x3a04af runnable
> [0x00007f2d878d0000]
> java.lang.Thread.State: RUNNABLE
> at org.apache.cassandra.dht.Range.isWrapAround(Range.java:260)
> at org.apache.cassandra.dht.Range.contains(Range.java:51)
> at org.apache.cassandra.dht.Range.contains(Range.java:110)
> at
> org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(TokenMetadata.java:916)
> at
> org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:775)
> at
> org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:541)
> at
> org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:616)
> at
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1101)
> at
> org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:1083)
> at
> org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:976)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3996)
> at
> org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3980)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:205)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
