I would try running it with memtable_offheap_space_in_mb at the default for sure, but definitely lower than 8GB. With 32GB of RAM, you're already allocating half of that for your heap, and then halving the remainder for off heap memtables. What's left may not be enough for the OS, etc. Giving some of that back, will allow more to be used for page cache, which always helps.
"JVM heap size: 16GB, CMS, 1GB newgen" For CMS GC with a 16GB heap, 1GB is way too small for new gen. You're going to want that to be at least 40% of the max heap size. Some folks here even advocate for setting Xmn as high as 50% of Xmx/s. If you want to stick with CMS GC, take a look at https://issues.apache.org/jira/browse/CASSANDRA-8150. There's plenty of good info in there on CMS GC tuning. Make sure to read through the whole ticket, so that you understand what each setting does. You can't just pick-and-choose. Regards, Aaron On Tue, Jun 2, 2020 at 1:31 AM onmstester onmstester <onmstes...@zoho.com.invalid> wrote: > I just changed these properties to increase flushed file size (decrease > number of compactions): > > - memtable_allocation_type from heap_buffers to offheap_objects > - memtable_offheap_space_in_mb: from default (2048) to 8192 > > Using default value for other memtable/compaction/commitlog configurations > . > > After a few hours some of nodes stopped to do any mutations (dropped > mutaion increased) and also pending flushes increased, they were just up > and running and there was only a single CPU core with 100% usage(other > cores was 0%). other nodes on the cluster determines the node as DN. Could > not access 7199 and also could not create thread dump even with jstack -F. > > Restarting Cassandra service fixes the problem but after a while some > other node would be DN. > > Am i missing some configurations? What should i change in cassandra > default configuration to maximize write throughput in single node/cluster > in write-heavy scenario for the data model: > Data mode is a single table: > create table test( > text partition_key, > text clustering_key, > set<text> rows, > primary key ((partition_key, clustering_key)) > > > vCPU: 12 > Memory: 32GB > Node data size: 2TB > Apache cassandra 3.11.2 > JVM heap size: 16GB, CMS, 1GB newgen > > Sent using Zoho Mail <https://www.zoho.com/mail/> > > > >
