[
https://issues.apache.org/jira/browse/CASSANDRA-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617137#comment-14617137
]
Benedict commented on CASSANDRA-9662:
-------------------------------------
This patch looks like it would acceptably fix the bouncing report of compaction
tasks outstanding.
However I think we should step back for a moment and consider the context. The
reason this is bouncing around is because we attempt to impose a constraint in
{{submitBackground}} that we enforce very badly. This leads us to accidentally
submit perhaps thousands of no-op compactions. This patch masks that from the
user, but we're still doing it. This constraint being enforced _at all_ is very
questionable, though. It can lead us to starving entire tables from compaction
for extended periods, because they happen to flush at periods during which the
compaction executors are fully occupied. Ironically, CASSANDRA-9592 both
mitigates and worsens this problem, depending on if you're a secondary index or
not (for whom it makes things worse; everyone else it should prevent starvation
for)
So: how about we always submit exactly one background task for any table
whenever one is requested. That way the numbers will never bounce around so
wildly: they'll be capped at a variance of about 1 per table, per flush or per
minute. We can (and perhaps should) also introduce this patch, but it would be
comparatively much less of a problem, and we would have fixed a potential
problematic situation where tables get very far behind on compaction.
> compactionManager reporting wrong pendingtasks
> ----------------------------------------------
>
> Key: CASSANDRA-9662
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9662
> Project: Cassandra
> Issue Type: Bug
> Components: API
> Environment: OS: Amazon Linux AMI release 2015.03
> Cassandra: 2.0.16
> JVM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25, mixed mode)
> Java: version 1.8.0_40, vendor Oracle Corporation
> CPU: 8 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
> Memory: 32G
> Reporter: Tony Xu
> Assignee: Yuki Morishita
> Priority: Minor
> Fix For: 2.0.x
>
> Attachments: node1.jpg
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Yesterday I upgraded my Cassandra cluster from 2.0.14 to 2.0.16, after
> upgrade, I am start seeing some strange behaviours of "PendingTasks"
> reporting.
> The Cassandra repository I am using is datastax, steps I performed for
> upgrade:
> yum update -y cassandra20
> The upgrade went fine, after upgrade cluster is operating okay. "nodetool
> info and nodetool status" results looked fine. "nodetool version" is
> reporting the correct version.
> But our monitoring system start reporting some crazy "pendingtasks". For
> example, "pending taks" for node1 sometimes jump from 0 to 15K for about 1
> minute, then drop back to 0. This issue keeps occurring, didn't have this
> issue with 2.0.14. Our monitoring system is checking the value of "MBeans" ->
> "CompactionManager" -> "PendingTasks".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
