Pending tasks are not being processed.
Hi all, Yesterday I put a lot of blobs into Cassandra and it created many, probably compaction, pending tasks (few hundreds according to Ops Center). On all nodes all pending tasks were eventually processed, but on one problematic, I see no related activity. Problematic node seems to be responsive and no errors in logs. What can be the reason? Cassandra ver: multiple versions on 14 nodes (2.0.8 and 2.0.9) OpsCenter ver: 4.1.2 Compaction type: leveled (we had capacity issues with size-tiered). To process pending task as fast as possible, I temporary changed compaction_throughput_mb_per_sec to 0 on this specific node. It helps, but only when compaction running, and currently pending tasks are not being processed. Thanks, Pavel
Re: Pending tasks are not being processed.
On Thu, Sep 4, 2014 at 8:09 AM, Pavel Kogan wrote: > Yesterday I put a lot of blobs into Cassandra and it created many, > probably compaction, pending tasks (few hundreds according to Ops Center). > On all nodes all pending tasks were eventually processed, but on one > problematic, I see no related activity. Problematic node seems to be > responsive and no errors in logs. What can be the reason? > > Cassandra ver: multiple versions on 14 nodes (2.0.8 and 2.0.9) > Running for extended periods of time with split versions is Not Supported. That said, perhaps you are running into... https://issues.apache.org/jira/browse/CASSANDRA-7145 or https://issues.apache.org/jira/browse/CASSANDRA-7808 ? =Rob
Re: Pending tasks are not being processed.
Should I experience any problems even if split versions vary only by minor digit? After another restart of node, it seems that the problem was solved somehow. Regards, Pavel On Thu, Sep 4, 2014 at 1:59 PM, Robert Coli wrote: > On Thu, Sep 4, 2014 at 8:09 AM, Pavel Kogan > wrote: > >> Yesterday I put a lot of blobs into Cassandra and it created many, >> probably compaction, pending tasks (few hundreds according to Ops Center). >> On all nodes all pending tasks were eventually processed, but on one >> problematic, I see no related activity. Problematic node seems to be >> responsive and no errors in logs. What can be the reason? >> >> Cassandra ver: multiple versions on 14 nodes (2.0.8 and 2.0.9) >> > > Running for extended periods of time with split versions is Not Supported. > > That said, perhaps you are running into... > > https://issues.apache.org/jira/browse/CASSANDRA-7145 > or > https://issues.apache.org/jira/browse/CASSANDRA-7808 > > ? > > =Rob > >
Re: Pending tasks are not being processed.
On Thu, Sep 4, 2014 at 11:54 AM, Pavel Kogan wrote: > Should I experience any problems even if split versions vary only by minor > digit? > My statement states what it states, no more and no less. It is Not Supported to run with split minor version, single digit increment or not, for longer than an upgrade takes. > After another restart of node, it seems that the problem was solved > somehow. > Fixed forever! =Rob http://twitter.com/rcolidba
