[
https://issues.apache.org/jira/browse/CASSANDRA-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350474#comment-14350474
]
T Jake Luciani commented on CASSANDRA-8692:
-------------------------------------------
bq. Are you testing with or without vnodes?
That was with vnodes. Actually without vnodes (on stock 2.1.3) I'm seeing
almost a 2x improvement in throughput and latency on this 99 node cluster.
This is pretty fundemental to vnodes as [~aweisberg] mentions. Without vnodes
we are always sending replica writes to the same N nodes and with vnodes we
split them up across all nodes in the cluster. The OS is naturally coalescing
these packets as you can see by the rAvs and wAvs columns below.
It seems like there is a much bigger issue here related to vnodes we need to
solve. perhaps a more controlled/coordinated allocation of vnodes so we get
behavior more like the no vnode case (which we naturally get massive
improvement in packet size).
Here are the network stats with vnodes:
{code}
Time Int rMbps wMbps rPk/s wPk/s rAvs wAvs %rUtil %wUtil
15:08:23 eth0 73.66 55.39 45115.6 44654.8 214.0 162.6 7.72 5.81
15:08:24 eth0 78.23 71.56 53299.8 52009.3 192.4 180.3 8.20 7.50
15:08:25 eth0 78.27 65.26 54357.8 53115.0 188.7 161.0 8.21 6.84
15:08:26 eth0 92.81 78.24 63269.4 61833.7 192.3 165.8 9.73 8.20
15:08:27 eth0 87.64 73.58 59189.0 57346.1 194.1 168.2 9.19 7.72
15:08:28 eth0 98.08 82.57 65408.3 63701.3 196.5 169.9 10.3 8.66
15:08:29 eth0 99.20 82.75 61070.9 60120.2 212.9 180.4 10.4 8.68
15:08:30 eth0 87.33 72.21 55217.8 53205.2 207.3 177.9 9.16 7.57
15:08:31 eth0 100.2 82.88 62752.4 60735.0 209.3 178.9 10.5 8.69
15:08:32 eth0 93.25 76.35 60180.5 58639.9 203.1 170.7 9.78 8.01
15:08:33 eth0 89.56 77.24 57428.4 56417.9 204.4 179.4 9.39 8.10
15:08:34 eth0 94.51 79.95 63685.5 63274.3 194.5 165.6 9.91 8.38
15:08:35 eth0 93.94 78.01 60504.5 59120.5 203.5 173.0 9.85 8.18
15:08:36 eth0 95.42 77.78 62392.6 60861.0 200.5 167.5 10.0 8.16
15:08:37 eth0 81.13 70.63 57439.5 56605.9 185.1 163.5 8.51 7.41
15:08:38 eth0 83.14 70.22 58104.0 56434.0 187.6 163.1 8.72 7.36
15:08:39 eth0 92.02 76.24 58900.5 55923.6 204.8 178.7 9.65 7.99
15:08:40 eth0 102.5 86.29 66515.7 66032.8 202.0 171.3 10.8 9.05
15:08:41 eth0 92.04 70.23 58642.6 56185.1 205.7 163.8 9.65 7.36
15:08:42 eth0 97.59 87.51 64721.8 64347.4 197.6 178.3 10.2 9.18
{code}
And without vnodes:
{code}
Time Int rMbps wMbps rPk/s wPk/s rAvs wAvs %rUtil %wUtil
14:46:24 eth0 133.6 111.2 35240.4 33946.6 496.8 429.3 14.0 11.7
14:46:25 eth0 113.4 91.75 30029.4 29208.0 495.1 411.7 11.9 9.62
14:46:26 eth0 123.4 96.78 33802.7 32350.2 478.4 392.1 12.9 10.1
14:46:27 eth0 120.4 95.19 32919.5 31669.6 479.4 394.0 12.6 9.98
14:46:28 eth0 119.2 89.46 32969.8 30548.0 474.0 383.8 12.5 9.38
14:46:29 eth0 149.7 123.2 34880.8 33658.9 562.4 479.9 15.7 12.9
14:46:30 eth0 133.2 106.0 30253.4 28806.6 576.9 482.5 14.0 11.1
14:46:31 eth0 124.6 94.99 30638.8 27731.3 532.8 448.9 13.1 9.96
14:46:32 eth0 128.7 104.0 30356.9 29852.5 555.7 456.7 13.5 10.9
14:46:33 eth0 127.8 101.2 31735.1 30715.3 527.7 432.0 13.4 10.6
14:46:34 eth0 133.4 107.2 32772.6 31283.3 533.5 449.1 14.0 11.2
14:46:35 eth0 112.2 88.93 35378.2 34920.8 415.7 333.8 11.8 9.33
14:46:36 eth0 128.4 103.3 36949.9 37194.1 455.5 363.9 13.5 10.8
14:46:37 eth0 104.2 82.93 30794.0 29068.2 443.7 374.0 10.9 8.70
14:46:38 eth0 103.8 78.08 30582.8 29560.7 445.0 346.2 10.9 8.19
14:46:39 eth0 136.1 112.1 29640.9 28823.1 601.6 509.7 14.3 11.8
14:46:40 eth0 138.0 110.5 32089.0 31454.5 563.6 460.6 14.5 11.6
14:46:41 eth0 129.5 103.9 32848.7 32135.9 516.6 423.9 13.6 10.9
14:46:42 eth0 110.0 85.14 33232.1 31965.6 433.7 349.1 11.5 8.93
14:46:43 eth0 134.8 109.3 33557.0 33590.9 526.7 426.5 14.1 11.5
{code}
> Coalesce intra-cluster network messages
> ---------------------------------------
>
> Key: CASSANDRA-8692
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8692
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Ariel Weisberg
> Assignee: Ariel Weisberg
> Fix For: 2.1.4
>
> Attachments: batching-benchmark.png
>
>
> While researching CASSANDRA-8457 we found that it is effective and can be
> done without introducing additional latency at low concurrency/throughput.
> The patch from that was used and found to be useful in a real life scenario
> so I propose we implement this in 2.1 in addition to 3.0.
> The change set is a single file and is small enough to be reviewable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
