[ https://issues.apache.org/jira/browse/CASSANDRA-16495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17315055#comment-17315055 ]
Ekaterina Dimitrova edited comment on CASSANDRA-16495 at 4/16/21, 5:28 PM: --------------------------------------------------------------------------- Thank you both! I just squashed the commits; pushed patches and submitted CI runs for 3.0, 3.11 and 4.0. The only difference is that Stage class was refactored in 4.0 but this doesn't affect a lot the patch. [3.0 patch|https://github.com/ekaterinadimitrova2/cassandra/commit/71d06d9b45d907687cef7e1cc6d6210f1b2079c9] | [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/583/] [3.11 patch|https://github.com/ekaterinadimitrova2/cassandra/commit/74a6cbd82bfd3c7177f4067bf63e96114dc0ba85] | [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/584/] [trunk patch|https://github.com/ekaterinadimitrova2/cassandra/pull/97/commits/ad531e827d4dd88e7d688e79aaa2045c9bb49eb] | [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/585/] was (Author: e.dimitrova): Thank you both! I just squashed the commits; pushed patches and submitted CI runs for 3.0, 3.11 and 4.0. The only difference is that Stage class was refactored in 4.0 but this doesn't affect a lot the patch. [3.0 patch|https://github.com/ekaterinadimitrova2/cassandra/commit/71d06d9b45d907687cef7e1cc6d6210f1b2079c9] | [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/583/] [3.11 patch|https://github.com/ekaterinadimitrova2/cassandra/commit/74a6cbd82bfd3c7177f4067bf63e96114dc0ba85] | [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/584/] [trunk patch|https://github.com/ekaterinadimitrova2/cassandra/pull/97/commits/ad531e827d4dd88e7d688e79aaa2045c9bb49eb7] | [CI run |https://jenkins-cm4.apache.org/job/Cassandra-devbranch/585/] > Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown > During Decommission > ------------------------------------------------------------------------------------------------ > > Key: CASSANDRA-16495 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16495 > Project: Cassandra > Issue Type: Bug > Components: Local/Startup and Shutdown > Reporter: Caleb Rackliffe > Assignee: Ekaterina Dimitrova > Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc1 > > > A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t > happen consistently. When [failure > occurs|https://app.circleci.com/pipelines/github/maedhroz/cassandra/235/workflows/eb8133ce-9373-4136-b404-ceca167353f6/jobs/1355/tests], > it appears to be because a delayed schema pull happens after decommission > shuts down the MIGRATION stage’s thread pool. > {noformat} > ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284 > CassandraDaemon.java:579 - Exception in thread > Thread[node1_NonPeriodicTasks:1,5,node1] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut > down at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) > > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176) > > at > java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) > at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129) > at > org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > > at java.base/java.lang.Thread.run(Thread.java:834) > {noformat} > A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks > in StorageService#decommission(). See the original discussion > [here|https://issues.apache.org/jira/browse/CASSANDRA-16181?focusedCommentId=17293329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17293329]. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org