[
https://issues.apache.org/jira/browse/CASSANDRA-8072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500545#comment-14500545
]
Brandon Williams edited comment on CASSANDRA-8072 at 4/19/15 6:39 PM:
--
After deep packet inspection, I believe I've found the root non-reconnectable
snitch part of this issue. When you decom a node, it never correctly tears
down its ITC pools, which leaves the other side with a dead OTC pool:
{noformat}
tcp1 0 10.208.8.123:33441 10.208.8.63:7000CLOSE_WAIT
18401/java
{noformat}
Now when you try to bootstrap with the same IP, the shadow syn is correctly
sent and the ack reply is built and queued, but MS tries to use the now defunct
OTC pool and the message never makes it back to the node, since it just sends
TCP RSTs which finally kills the connection. But since the gossip syn is only
sent once, the seed has nothing else to send the node and never reestablishes
the connection, leaving the bootstrapping node thinking it never talked to a
seed and throwing this error.
was (Author: brandon.williams):
After deep packet inspection, I believe I've found the root non-reconnectable
snitch part of this issue. When you decom a node, it never correctly tears
down its ITC pools, which leaves the other side with a dead OTC pool:
{noformat}
tcp1 0 10.208.8.123:33441 10.208.8.63:7000CLOSE_WAIT
18401/java
{noformat}
Now when you try to bootstrap with the same IP, the shadow syn is correctly
sent and the ack reply is built and queued, but MS tries to use the now default
OTC pool and the message never makes it back to the node, since it just sends
RSTs which finally kills the connection. But since the syn is only sent once,
the seed has nothing else to send the node and never reestablishes the
connection, leaving the bootstrapping node thinking it never talked to a seed
and throwing this error.
Exception during startup: Unable to gossip with any seeds
-
Key: CASSANDRA-8072
URL: https://issues.apache.org/jira/browse/CASSANDRA-8072
Project: Cassandra
Issue Type: Bug
Reporter: Ryan Springer
Assignee: Brandon Williams
Fix For: 2.0.15, 2.1.5
Attachments: cas-dev-dt-01-uw1-cassandra-seed01_logs.tar.bz2,
cas-dev-dt-01-uw1-cassandra-seed02_logs.tar.bz2,
cas-dev-dt-01-uw1-cassandra02_logs.tar.bz2,
casandra-system-log-with-assert-patch.log, trace_logs.tar.bz2
When Opscenter 4.1.4 or 5.0.1 tries to provision a 2-node DSC 2.0.10 cluster
in either ec2 or locally, an error occurs sometimes with one of the nodes
refusing to start C*. The error in the /var/log/cassandra/system.log is:
ERROR [main] 2014-10-06 15:54:52,292 CassandraDaemon.java (line 513)
Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1200)
at
org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:444)
at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:655)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:609)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:502)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
INFO [StorageServiceShutdownHook] 2014-10-06 15:54:52,326 Gossiper.java
(line 1279) Announcing shutdown
INFO [StorageServiceShutdownHook] 2014-10-06 15:54:54,326
MessagingService.java (line 701) Waiting for messaging service to quiesce
INFO [ACCEPT-localhost/127.0.0.1] 2014-10-06 15:54:54,327
MessagingService.java (line 941) MessagingService has terminated the accept()
thread
This errors does not always occur when provisioning a 2-node cluster, but
probably around half of the time on only one of the nodes. I haven't been
able to reproduce this error with DSC 2.0.9, and there have been no code or
definition file changes in Opscenter.
I can reproduce locally with the above steps. I'm happy to test any proposed
fixes since I'm the only person able to reproduce reliably so far.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)