[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522135#comment-17522135 ] Stefan Miklosovic commented on CASSANDRA-17493: --- https://app.circleci.com/pipelines/github/instaclustr/cassandra/919/workflows/090bb620-b9a8-40f4-a83c-b5d7cfd8e5c4 > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 2h > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521814#comment-17521814 ] Jon Meredith commented on CASSANDRA-17493: -- Thanks for responding to feedback. +1 on the most recent version provided you get an explainably clean test run. > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 1h 50m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521148#comment-17521148 ] Stefan Miklosovic commented on CASSANDRA-17493: --- [https://app.circleci.com/pipelines/github/instaclustr/cassandra/915/workflows/ac6f220f-a346-4728-8bcd-6ccc711f6951] https://github.com/apache/cassandra/pull/1563 > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520611#comment-17520611 ] Stefan Miklosovic commented on CASSANDRA-17493: --- [~jonmeredith] sure, dont rush it though, in the meanwhile I hope [~paulo] takes the second look too. I ll wait till ... Wednesday morning UTC+2. > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520609#comment-17520609 ] Stefan Miklosovic commented on CASSANDRA-17493: --- solution with rewriting jvm dtest to plain unit test here (1), build here (2) (1) https://github.com/apache/cassandra/pull/1563 (2) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1593/ > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520595#comment-17520595 ] Jon Meredith commented on CASSANDRA-17493: -- [~smiklosovic] I can take a look at this today as well if you'd like extra eyes on it. > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520569#comment-17520569 ] Stefan Miklosovic commented on CASSANDRA-17493: --- [~paulo] here https://github.com/apache/cassandra/blob/trunk/test/distributed/org/apache/cassandra/distributed/impl/Instance.java#L756 > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520565#comment-17520565 ] Stefan Miklosovic commented on CASSANDRA-17493: --- [~paulo] it is because the test is in fact draining twice - the first time I drain in the test itself - programmatically. The second time the draining process is executed when Cassandra node itself is going down as part of closing the cluster. It will fail - it will throw - that it was already shut down - and it was - but by myself as part of testing it. > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[
https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520546#comment-17520546
]
Paulo Motta commented on CASSANDRA-17493:
-
Is this error [found
here|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1590/testReport/org.apache.cassandra.distributed.test/CasCriticalSectionTest/criticalSectionTest/]
related to this change? I don't think so but just checking.
{code:none}
ERROR 22:18:29 Exception in thread Thread[MutationStage-1,5,SharedPool]
java.lang.RuntimeException: java.lang.IllegalStateException: HintsService is
shut down and can't accept new hints
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2577)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:120)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: HintsService is shut down and can't
accept new hints
at org.apache.cassandra.hints.HintsService.write(HintsService.java:165)
at
org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:2656)
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2573)
... 6 common frames omitted
ERROR [MutationStage-1] node1 2022-04-09 22:18:29,622
JVMStabilityInspector.java:68 - Exception in thread
Thread[MutationStage-1,5,SharedPool]
java.lang.RuntimeException: java.lang.IllegalStateException: HintsService is
shut down and can't accept new hints
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2577)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:81)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:120)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: HintsService is shut down and can't
accept new hints
at org.apache.cassandra.hints.HintsService.write(HintsService.java:165)
at
org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:2656)
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2573)
... 6 common frames omitted
{code}
> Shutdown all ScheduledExecutors as part of node drainage
>
>
> Key: CASSANDRA-17493
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17493
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are currently shutting down only non-periodic executors in
> StorageService#drain. We should shut down all of them. As of now, there does
> not seem to be any reason why these executors should be active.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[
https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520529#comment-17520529
]
Paulo Motta commented on CASSANDRA-17493:
-
Looks good to me - can you just clarify why this was changed?
{code:java}
if (isShutDown)
-throw new IllegalStateException("HintsService has already been
shut down");
+{
+logger.warn("HintsService has already been shut down");
+return;
+}
{code}
> Shutdown all ScheduledExecutors as part of node drainage
>
>
> Key: CASSANDRA-17493
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17493
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are currently shutting down only non-periodic executors in
> StorageService#drain. We should shut down all of them. As of now, there does
> not seem to be any reason why these executors should be active.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520442#comment-17520442 ] Stefan Miklosovic commented on CASSANDRA-17493: --- [~brandon.williams] would you take a look at this one please? This is basically a blocker for 17180. I desperately want to see all this stuff in 4.1. > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514114#comment-17514114 ] Stefan Miklosovic commented on CASSANDRA-17493: --- [~sathyakplm] We are implementing CASSANDRA-17180 where we want to start a periodic task and we want to be sure that such task will not be executed on drain and we do not want to hold any Future reference to it or anything like that. So we are relying on the fact that periodic task executors are stopped as part of the drain which is not happening now. > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug >Reporter: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514092#comment-17514092 ] Sathyanarayanan Saravanamuthu commented on CASSANDRA-17493: --- Hi [~smiklosovic], why is this a bug? What benefits do the changes in the PR bring in? > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug >Reporter: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17493) Shutdown all ScheduledExecutors as part of node drainage
[ https://issues.apache.org/jira/browse/CASSANDRA-17493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513597#comment-17513597 ] Stefan Miklosovic commented on CASSANDRA-17493: --- https://github.com/apache/cassandra/pull/1529 > Shutdown all ScheduledExecutors as part of node drainage > > > Key: CASSANDRA-17493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17493 > Project: Cassandra > Issue Type: Bug >Reporter: Stefan Miklosovic >Priority: Normal > Fix For: 4.x > > > We are currently shutting down only non-periodic executors in > StorageService#drain. We should shut down all of them. As of now, there does > not seem to be any reason why these executors should be active. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
