Is this on 4.0.0 ? 4.0.1 fixes an issue where the gossip result is too large for the urgent message queue, causing this stack trace, and was released 3 days ago. I've never seen it on a 10 node cluster before, but I'd be trying that.
On Fri, Sep 10, 2021 at 7:50 AM Joe Obernberger < joseph.obernber...@gmail.com> wrote: > I have a 10 node cluster and am trying to add another node. The new > node is running Rocky Linux and I'm getting the unable to gossip with > any peers error. Firewall and SELinux are off. I can ping all the > other nodes OK. I've checked everything I can think of (/etc/hosts, > listen_address, broadcast etc..). It all looks correct to me. > Any ideas? Could it be an incompatibility with Rocky? > > DEBUG [main] 2021-09-10 06:45:24,846 YamlConfigurationLoader.java:112 - > Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml > INFO [Messaging-EventLoop-3-6] 2021-09-10 06:45:24,921 > OutboundConnection.java:1150 - > /172.16.100.44:7000(/172.16.100.44:45934)->/172.16.100.253:7000-URGENT_MESSAGES-90efbb9e > > successfully connected, version = 12, framing = LZ4, encryption = > unencrypted > INFO [Messaging-EventLoop-3-3] 2021-09-10 06:45:24,930 > OutboundConnection.java:1150 - > /172.16.100.44:7000(/172.16.100.44:44320)->/172.16.100.37:7000-URGENT_MESSAGES-eae47864 > > successfully connected, version = 12, framing = LZ4, encryption = > unencrypted > INFO [ScheduledTasks:1] 2021-09-10 06:45:27,648 TokenMetadata.java:525 > - Updating topology for all endpoints that have changed > DEBUG [OptionalTasks:1] 2021-09-10 06:45:54,644 > SizeEstimatesRecorder.java:65 - Node is not part of the ring; not > recording size estimates > ERROR [main] 2021-09-10 06:46:25,891 CassandraDaemon.java:909 - > Exception encountered during startup > java.lang.RuntimeException: Unable to gossip with any peers > at > org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1805) > at > > org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:648) > at > > org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:934) > at > > org.apache.cassandra.service.StorageService.initServer(StorageService.java:784) > at > > org.apache.cassandra.service.StorageService.initServer(StorageService.java:729) > at > > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420) > at > > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:763) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:887) > DEBUG [StorageServiceShutdownHook] 2021-09-10 06:46:25,896 > StorageService.java:1621 - DRAINING: starting drain process > INFO [StorageServiceShutdownHook] 2021-09-10 06:46:25,898 > HintsService.java:220 - Paused hints dispatch > WARN [StorageServiceShutdownHook] 2021-09-10 06:46:25,899 > Gossiper.java:1993 - No local state, state is in silent shutdown, or > node hasn't joined, not announcing shutdown > > Thank you! > > -Joe > >
