Kurt, in our test it also didn’t made a difference with the default number of GC Threads (43 on our large machine) and running with Xmx128M or XmX31G (derived from $MAX_HEAP_SIZE). For both Xmx, we saw the high CPU caused by nodetool.
Regards, Thomas From: kurt greaves [mailto:k...@instaclustr.com] Sent: Dienstag, 29. Mai 2018 13:06 To: User <user@cassandra.apache.org> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage Thanks Thomas. After a bit more research today I found that the whole $MAX_HEAP_SIZE issue isn't really a problem because we don't explicitly set -Xms so the minimum heapsize by default will be 256mb, which isn't hugely problematic, and it's unlikely more than that would get allocated. On 29 May 2018 at 09:29, Steinmaurer, Thomas <thomas.steinmau...@dynatrace.com<mailto:thomas.steinmau...@dynatrace.com>> wrote: Hi Kurt, thanks for pointing me to the Xmx issue. JIRA + patch (for Linux only based on C* 3.11) for the parallel GC thread issue is available here: https://issues.apache.org/jira/browse/CASSANDRA-14475 Thanks, Thomas From: kurt greaves [mailto:k...@instaclustr.com<mailto:k...@instaclustr.com>] Sent: Dienstag, 29. Mai 2018 05:54 To: User <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: nodetool (2.1.18) - Xmx, ParallelGCThreads, High CPU usage 1) nodetool is reusing the $MAX_HEAP_SIZE environment variable, thus if we are running Cassandra with e.g. Xmx31G, nodetool is started with Xmx31G as well This was fixed in 3.0.11/3.10 in CASSANDRA-12739<https://issues.apache.org/jira/browse/CASSANDRA-12739>. Not sure why it didn't make it into 2.1/2.2. 2) As -XX:ParallelGCThreads is not explicitly set upon startup, this basically defaults to a value dependent on the number of cores. In our case, with the machine above, the number of parallel GC threads for the JVM is set to 43! 3) Test-wise, we have adapted the nodetool startup script in a way to get a Java Flight Recording file on JVM exit, thus with each nodetool invocation we can inspect a JFR file. Here we may have seen System.gc() calls (without visible knowledge where they come from), GC times for the entire JVM life-time (e.g. ~1min) showing high cpu. This happened for both Xmx128M (default as it seems) and Xmx31G After explicitly setting -XX:ParallelGCThreads=1 in the nodetool startup script, CPU usage spikes by nodetool are entirely gone. Is this something which has been already adapted/tackled in Cassandra versions > 2.1 or worth to be considered as some sort of RFC? Can you create a JIRA for this (and a patch, if you like)? We should be explicitly setting this on nodetool invocations. The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313 The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313