[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14140007#comment-14140007
]
graham sanderson commented on CASSANDRA-7546:
---------------------------------------------
Didn't want to deep dive, but out of curiosity I did do one run configured for
a single partition
{code}
Results:
op rate : 5760
partition rate : 5760
row rate : 5760
latency mean : 158.7
latency median : 151.2
latency 95th percentile : 221.5
latency 99th percentile : 262.3
latency 99.9th percentile : 282.4
latency max : 396.0
total gc count : 3
total gc mb : 18779
total gc time (s) : 0
avg gc time(ms) : 67
stdev gc time(ms) : 26
Total operation time : 00:00:35
Improvement over 609 threadCount: 4%
id, total ops , adj row/s, op/s, pk/s, row/s, mean,
med, .95, .99, .999, max, time, stderr, gc: #, max ms,
sum ms, sdv ms, mb
4 threadCount, 6782 , -0, 120, 120, 120, 33.3,
43.8, 50.6, 63.0, 83.9, 85.7, 56.6, 0.01940, 0, 0,
0, 0, 0
8 threadCount, 6629 , -0, 212, 212, 212, 37.7,
39.1, 57.0, 75.0, 127.2, 138.2, 31.3, 0.00868, 0, 0,
0, 0, 0
16 threadCount, 27730 , -0, 566, 566, 566, 28.2,
26.2, 50.6, 75.7, 125.5, 170.4, 49.0, 0.01963, 0, 0,
0, 0, 0
24 threadCount, 51763 , 798, 796, 796, 796, 30.1,
29.5, 51.0, 76.9, 90.8, 144.4, 65.0, 0.01977, 2, 203,
203, 10, 12877
36 threadCount, 74953 , -0, 1253, 1253, 1253, 28.7,
27.8, 50.7, 60.5, 79.6, 308.0, 59.8, 0.01938, 0, 0,
0, 0, 0
54 threadCount, 56948 , -0, 1807, 1807, 1807, 29.8,
27.6, 52.6, 63.1, 78.1, 121.1, 31.5, 0.01170, 3, 176,
176, 12, 19816
81 threadCount, 74856 , -0, 2369, 2369, 2369, 34.1,
33.2, 57.2, 67.6, 76.6, 108.6, 31.6, 0.00946, 0, 0,
0, 0, 0
121 threadCount, 100526 , -0, 3158, 3158, 3158, 38.2,
37.8, 63.4, 78.9, 89.1, 446.6, 31.8, 0.01805, 2, 93,
93, 1, 13063
181 threadCount, 277875 , -0, 4491, 4491, 4491, 40.2,
40.2, 63.1, 79.1, 94.0, 679.7, 61.9, 0.01985, 5, 286,
286, 28, 32541
271 threadCount, 169870 , -0, 5205, 5205, 5205, 52.0,
49.2, 84.9, 110.5, 140.5, 843.9, 32.6, 0.01320, 3, 157,
157, 11, 19408
406 threadCount, 187985 , 5648, 5555, 5555, 5555, 73.0,
64.2, 122.1, 156.0, 285.3, 848.6, 33.8, 0.01421, 3, 173,
173, 12, 19570
609 threadCount, 201184 , 5540, 5534, 5534, 5534, 110.1,
101.1, 160.5, 230.1, 378.9, 555.6, 36.4, 0.01917, 3, 163,
163, 17, 19709
913 threadCount, 205466 , 5787, 5760, 5760, 5760, 158.7,
151.2, 221.5, 262.3, 282.4, 396.0, 35.7, 0.01335, 3, 200,
200, 26, 18779
{code}
Obviously I don't know if the slowdown is on the load end or the server end
(though there is some GC increase here - we'll see what the patch for this
issue does). Note that if this is a synchronization problem with the load
generator still, we do know for a fact that hinting is a good way of turning a
large partition domain into a small partition domain (so I'll obviously be
testing that too, though that isn't apples to apples either).
> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-7546
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: graham sanderson
> Assignee: graham sanderson
> Fix For: 2.1.1
>
> Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt,
> 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt,
> 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png,
> suggestion1.txt, suggestion1_21.txt, young_gen_gc.png
>
>
> In order to preserve atomicity, this code attempts to read, clone/update,
> then CAS the state of the partition.
> Under heavy contention for updating a single partition this can cause some
> fairly staggering memory growth (the more cores on your machine the worst it
> gets).
> Whilst many usage patterns don't do highly concurrent updates to the same
> partition, hinting today, does, and in this case wild (order(s) of magnitude
> more than expected) memory allocation rates can be seen (especially when the
> updates being hinted are small updates to different partitions which can
> happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation
> whilst not slowing down the very common un-contended case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
