Hi Pierre,
Is there anything stopping you from using the compaction_throughput
<https://github.com/apache/cassandra/blob/f9e033f519c14596da4dc954875756a69aea4e78/conf/cassandra.yaml#L989>
option in the cassandra.yaml file to manage the performance impact of
compaction operations?
With thread priority, there's a failure scenario on busy nodes when the
read operations uses too much CPU. If the compaction thread has lower
priority, it does not get enough CPU time to run, and SSTable files will
build up, causing read to become slower and more expensive, which in
turn result in compaction gets even less CPU time. At the end, one of
the following three will happen:
* the node becomes too slow and most queries time out
* the Java process crashes due to too many open files or OOM because
JVM GC can't keep up
* the filesystem run out of free space or inodes
However, I'm unsure whether the compaction thread priority was
intentionally removed from 4.1.0. Someone familiar with this matter may
be able to answer that.
Cheers,
Bowen
On 22/02/2024 13:26, Pierre Fersing wrote:
Hello all,
I've recently upgraded to Cassandra 4.1 and see a change in compaction
behavior that seems unwanted:
* With Cassandra 3.11 compaction was run by thread in low priority and
thus using CPU nice (visible using top) (I believe Cassandra 4.0 also
had this behavior)
* With Cassandra 4.1, compactions are no longer run as low priority
thread (compaction now use "normal" CPU).
This means that when the server had limited CPU, Cassandra compaction
now compete for the CPU with other process (probably including
Cassandra itself) that need CPU. When it was using CPU nice, the
compaction only competed for CPU with other lower priority process
which was great as it leaves CPU available for processes that need to
kept small response time (like an API used by human).
Is it wanted to lose this feature in Cassandra 4.1 or was it just a
forget during re-write of compaction executor ? Should I open a bug to
re-introduce this feature in Cassandra ?
I've done few searches, and:
* I believe compaction used CPU nice because the compactor executor
was created with minimal priority:
https://github.com/apache/cassandra/blob/cassandra-3.11.16/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1906
<https://github.com/apache/cassandra/blob/cassandra-3.11.16/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L1906>
* I think it was dropped by commit
https://github.com/apache/cassandra/commit/be1f050bc8c0cd695a42952e3fc84625ad48d83a
<https://github.com/apache/cassandra/commit/be1f050bc8c0cd695a42952e3fc84625ad48d83a>
* It looks doable to set the thread priority with new executor, I
think adding ".withThreadPriority(Thread.MIN_PRIORITY)" when using
executorFactory in
https://github.com/apache/cassandra/blob/77a3e0e818df3cce45a974ecc977ad61bdcace47/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L2028
<https://github.com/apache/cassandra/blob/77a3e0e818df3cce45a974ecc977ad61bdcace47/src/java/org/apache/cassandra/db/compaction/CompactionManager.java#L2028>should
do it.
Did I miss a reason to no longer use low priority threads for
compaction ? Should I open a bug for re-adding this feature / submit a
PR ?
Regards,
Pierre Fersing
