Hi Jeff,
The cluster was build with 2.1 and has been upgraded to 3.11. It's using
"num_token: 64" and RF=3 for all keyspaces in all DCs.There are over 100 nodes
in each of 6 rings.I don't see any messages of the repair/steaming in
system.log so this shouldn't be related with the repair.I can
Hi Bowen,
There are many nodes having this issue and some of them repeatedly having
it.Replacing a node by wiping out everything and streaming in good shape of
sstables would work, but if we don't know the root cause the node would be in
the bad shape again.Yes, we know the reaper repair
Hi Bowen,
We do have reaper repair job scheduled periodically and it can take days even
weeks to complete one round of repair due to large number of rings/nodes.
However, we have paused the repair since we are facing this issue.We do not use
the MV in this cluster.There is major table taking
Hi Jiayong,
I'm sorry to hear that. I did not know many nodes were/are experiencing
the same issue. A bit of dig in the source code indicates the log below
comes from the ColumnFamilyStore.logFlush() method.
DEBUG [NativePoolCleaner] ColumnFamilyStore.java:932 -
Enqueuing flush of
A very large cluster using vnodes will cause lots of small sstables to
stream in during repair if the cluster is out of sync. This is one of the
reasons that the default number of vnodes was decreased in 4.0. How many
nodes in the cluster, how many DCs, how many vnodes per node, and how many
Hi Jiayong,
That doesn't really match the situation described in the SO question. I
suspected it was related to repairing a table with MV and large
partitions, but based on the information you've given, I was clearly wrong.
A few hundreds MB partitions is not exactly unusual, I don't see
Hi Jiayong,
Sorry I didn't make it clear in my previous email. When I commented on
the RAID0 setup, it was only a comment on the RAID0 setup vs JBOD, and
that was not in relation to the SSTable flushing issue. The part of my
previous email after the "On the frequent SSTable flush issue" line
