[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14150409#comment-14150409 ]
graham sanderson edited comment on CASSANDRA-7546 at 9/27/14 4:56 AM: ---------------------------------------------------------------------- Busy week - I I just did native_objects runs. It actually really helps out here a lot too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graph2_7546.png! was (Author: graham sanderson): Busy week - I I just did native_objects runs. It actually really helps out here a lot too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graphs2_7546.png! > AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory > ----------------------------------------------------------------------------- > > Key: CASSANDRA-7546 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: graham sanderson > Assignee: graham sanderson > Fix For: 2.1.1 > > Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, > 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, > 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, > graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, > young_gen_gc.png > > > In order to preserve atomicity, this code attempts to read, clone/update, > then CAS the state of the partition. > Under heavy contention for updating a single partition this can cause some > fairly staggering memory growth (the more cores on your machine the worst it > gets). > Whilst many usage patterns don't do highly concurrent updates to the same > partition, hinting today, does, and in this case wild (order(s) of magnitude > more than expected) memory allocation rates can be seen (especially when the > updates being hinted are small updates to different partitions which can > happen very fast on their own) - see CASSANDRA-7545 > It would be best to eliminate/reduce/limit the spinning memory allocation > whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)