Hi , I'll recommend tuning you heap size further( preferably lower) as large Heap size can lead to Large Garbage collection pauses also known as also known as a stop-the-world event. A pause occurs when a region of memory is full and the JVM needs to make space to continue. During a pause all operations are suspended. Because a pause affects networking, the node can appear as down to other nodes in the cluster. Additionally, any Select and Insert statements will wait, which increases read and write latencies.
Any pause of more than a second, or multiple pauses within a second that add to a large fraction of that second, should be avoided. The basic cause of the problem is the rate of data stored in memory outpaces the rate at which data can be removed MUTATION : If a write message is processed after its timeout (write_request_timeout_in_ms) it either sent a failure to the client or it met its requested consistency level and will relay on hinted handoff and read repairs to do the mutation if it succeeded. Another possible cause of the Issue could be you HDDs as that could too be a bottleneck. *MAX_HEAP_SIZE* The recommended maximum heap size depends on which GC is used: Hardware setupRecommended MAX_HEAP_SIZE Older computers Typically 8 GB. CMS for newer computers (8+ cores) with up to 256 GB RAM No more 14 GB. Thanks, Hitesh dua hiteshd...@gmail.com On Wed, Apr 18, 2018 at 10:07 PM, shalom sagges <shalomsag...@gmail.com> wrote: > Hi All, > > I have a 44 node cluster (22 nodes on each DC). > Each node has 24 cores and 130 GB RAM, 3 TB HDDs. > Version 2.0.14 (soon to be upgraded) > ~10K writes per second per node. > Heap size: 8 GB max, 2.4 GB newgen > > I deployed Reaper and GC started to increase rapidly. I'm not sure if it's > because there was a lot of inconsistency in the data, but I decided to > increase the heap to 16 GB and new gen to 6 GB. I increased the max tenure > from 1 to 5. > > I tested on a canary node and everything was fine but when I changed the > entire DC, I suddenly saw a lot of dropped mutations in the logs on most of > the nodes. (Reaper was not running on the cluster yet but a manual repair > was running). > > Can the heap increment cause lots of dropped mutations? > When is a mutation considered as dropped? Is it during flush? Is it during > the write to the commit log or memtable? > > Thanks! > > > >
