On Fri, May 5, 2017 at 1:20 PM Alain RODRIGUEZ <arodr...@gmail.com> wrote:
> Sorry to hear the restart did not help.
>
Hi,
We are hitting the same issue since a few weeks on version 3.0.16.
Normally, restarting an affected node helps, but this is something we would
like to avoid doing.
What makes it worse for us is that Cassandra Reaper stops scheduling new
repair jobs if it sees that the node have more than 20 pending compaction
tasks. We could bump this threshold, but in general the estimate could be
more accurate (or the actual tasks should be started timely).
Maybe try to monitor through JMX with
'org.apache.cassandra.db:type=CompactionManager',
>> attribute 'Compactions' or 'CompactionsSummary'
>
>
> What is this attribute showing?
>
For example, I have right now a node showing "pending tasks: 16" and no
compaction running. Here is the JMX output (well, via Jolokia):
"Compactions": [],
"CoreCompactorThreads": 1,
"CompactionSummary": [],
"MaximumCompactorThreads": 1,
"CoreValidationThreads": 1,
"MaximumValidatorThreads": 2147483647,
Here is the Apache Cassandra Jira:
> https://issues.apache.org/jira/browse/CASSANDRA. You search here
> <https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC>
> (
> https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC),
> for example.
>
I believe this is a separate issue. There some actual compaction tasks are
running, but not making progress. And we never TRUNCATEd our tables, and
for sure not recently.
Any more pointers on how to debug this?
Regards,
--
Alex
