[ https://issues.apache.org/jira/browse/CASSANDRA-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152387#comment-17152387 ]
Jai Bheemsen Rao Dhanwada commented on CASSANDRA-15592: ------------------------------------------------------- Hello [~brandon.williams] I ran into the similar Exception, is there any impact of this ERROR or this is just more of logging problem? in my tests I didn't see any impact to the cluster operations. so I would like to know the impact of this before even attempting to upgrade in production > IllegalStateException in gossip after removing node > --------------------------------------------------- > > Key: CASSANDRA-15592 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15592 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip > Reporter: Marcus Olsson > Assignee: Marcus Olsson > Priority: Normal > Fix For: 3.0.21, 3.11.7, 4.0, 4.0-alpha4 > > > In one of our test environments we encountered the following exception: > {noformat} > 2020-02-02T10:50:13.276+0100 [GossipTasks:1] ERROR > o.a.c.u.NoSpamLogger$NoSpamLogStatement:97 log > java.lang.IllegalStateException: Attempting gossip state mutation from > illegal thread: GossipTasks:1 > at > org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) > at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) > at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) > at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) > at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) > at > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > java.lang.IllegalStateException: Attempting gossip state mutation from > illegal thread: GossipTasks:1 > at > org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) > [apache-cassandra-3.11.5.jar:3.11.5] > at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) > [apache-cassandra-3.11.5.jar:3.11.5] > at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) > [apache-cassandra-3.11.5.jar:3.11.5] > at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) > [apache-cassandra-3.11.5.jar:3.11.5] > at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) > [apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) > [apache-cassandra-3.11.5.jar:3.11.5] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_231] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [na:1.8.0_231] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [na:1.8.0_231] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [na:1.8.0_231] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [na:1.8.0_231] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_231] > at > org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) > [apache-cassandra-3.11.5.jar:3.11.5] > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > ~[netty-all-4.1.42.Final.jar:4.1.42.Final] > at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231] > {noformat} > Since CASSANDRA-15059 we check that all state changes are performed in the > GossipStage but it seems like it was still performed in the "current" thread > [here|https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/gms/Gossiper.java#L895]. > It should be as simple as adding a > {code:java} > runInGossipStageBlocking(() ->) > {code} > for it. > I'll upload patches for 3.0, 3.11 and 4.0. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org