[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782218#comment-13782218
]
Aleksey Yeschenko commented on CASSANDRA-5932:
----------------------------------------------
First, let me thank you for your continued digging. Some of it helped. That
said, you should probably look at the current cassandra-2.0 branch, and not the
2.0.0 tarball/branches here in the comments.
bq. Issue 1 – When handling DigestMismatchException in
StorageProxy.fetchRows(), all data read requests are sent out using sendRR
without distinguishing remote nodes from the local node.
This is not an issue, and it's not spec retry related. Using LRR for local read
requests is merely an optimisation - there is nothing wrong with sendRR (not
that it isn't worth optimising here - just noting that it's not an issue). This
is also the answer to "How do we handle the case for local node? Does the
sendRR() and the corresponding receive part can handle the case for local node?
If not, then this may block for 10 seconds." Same goes for Issue 2 and Issue 3.
bq. The data read request for local node may never sent out. As one of the
nodes is down (which triggered the Speculative Retry) will cause one missing
response.
The former is not true, the latter won't, since the current cassandra-2.0 code
will send requests to all the contacted replicas. So if a node triggered spec
retry, that extra speculated replica will get the request as well, and we can
still satisfy the CL.
{noformat}
for (InetAddress endpoint : exec.getContactedReplicas())
{
Tracing.trace("Enqueuing full data read to {}",
endpoint);
MessagingService.instance().sendRR(message, endpoint,
repairHandler);
}
{noformat}
bq. Question for the Randomized approach – Since the end points are randomized,
the first node in the list is no likely the local node. This may cause a higher
possibility of data repair.
I don't see how the possibility of data repair is correlated with the locality
of a target node, but, it doesn't matter. The 'randomised approach' was an
experiment, it wasn't committed as part of the fix. See the latest
cassandra-2.0 branch code.
bq. In the Randomized Approach, the end points are reshuffled. Then, the first
node in the list used for data read request is not likely the local node. If
this node happens to be the DOWN node, then, we end with all digest responses
without the data, which will block and eventually timed out.
See the above reply.
TLDR: None of these seem to be issues, but we could optimise RR to use LRR for
local reads to get slightly better performance for local requests (and to be
consistent with the regular reads code).
> Speculative read performance data show unexpected results
> ---------------------------------------------------------
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
> Issue Type: Bug
> Reporter: Ryan McGuire
> Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932-6692c50412ef7d.png, 5932.ded39c7e1c2fa.logs.tar.gz,
> 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz,
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png,
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png,
> compaction-makes-slow-stats.png, eager-read-looks-promising.png,
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png,
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show
> undesirable behavior. I'm grouping these behaviours into one ticket as they
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 30000000 -n 30000000 -i 5 -l 2 -K
> 20
> node0 reads some data: stress -d node4 -n 30000000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance
> as the snitch adapts, but then it recovers... to even higher performance than
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they
> never recover their op-rate from before the compaction as the red and blue
> lines do.
> [Data for this test
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry,
> the green and orange line, at 450s showed the smallest dip in performance.
> [Data for this test
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager
> retry (red line) did poorly this time at 450s.
> [Data for this test
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
