A high level of compaction seems highly likely to throttle you by sending the service into a GC death spiral, doubly-so if any repairs happen to be underway at the same time (I may or may not have killed a few nodes this way, but I admit nothing!). Even if not in GC hell, it can cause you to episodically blast out writes that rapidly dirty a lot of pages, thus triggering a fill of the disk io queue that then starves out read requests from the disk. More != Better when it comes to compaction. You want as little compaction as your usage pattern requires of you. Smoothness of its contribution to the overall load is a better objective.
Jon Haddad did a datastax conference talk this year on some easy tunings that you’ll likely want to listen to. You’ll probably end up rethinking your vnode count as well. Also note that a fast disk can spend a lot of its time doing the wrong things. His talk covers some of the factors in that. https://www.youtube.com/watch?v=swL7bCnolkU From: "Steinmaurer, Thomas" <thomas.steinmau...@dynatrace.com> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> Date: Tuesday, October 22, 2019 at 6:47 AM To: "user@cassandra.apache.org" <user@cassandra.apache.org> Subject: Cassandra 2.1.18 - Question on stream/bootstrap throughput Message from External Sender Hello, using 2.1.8, 3 nodes (m4.10xlarge, ESB SSD-based), vnodes=256, RF=3, we are trying to add a 4th node. The two options to my knowledge, mainly affecting throughput, namely stream output and compaction throttling has been set to very high values (e.g. stream output = 800 Mbit/s resp. compaction throughput = 500 Mbyte/s) or even set to 0 (unthrottled) in cassandra.yaml + process restart. In both scenarios (throttling with high values vs. unthrottled), the 4th node is streaming from one node capped ~ 180-200Mbit/s, according to our SFM. The nodes have plenty of resources available (10Gbit, disk io/iops), also confirmed by e.g. iperf in regard to NW throughput and write to / read from disk in the area of 200 MByte/s. Are there any other known throughput / bootstrap limitations, which basically outrule above settings? Thanks, Thomas The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
