[ https://issues.apache.org/jira/browse/CASSANDRA-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914977#comment-13914977 ]
Brandon Williams commented on CASSANDRA-6772: --------------------------------------------- This should be easy enough to reproduce using a firewall to suddenly split the cluster and rejoin it later, but as far as I can tell it doesn't. > Cassandra inter data center communication broken > ------------------------------------------------ > > Key: CASSANDRA-6772 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6772 > Project: Cassandra > Issue Type: Bug > Environment: CentOS 6.0 > Reporter: Ananthkumar K S > Priority: Blocker > > I have two data enters DC1 and DC2. Both communicate via a private link. > Yesterday, we had a problem with a private link for 10 mins. From the time > the problem was resolved, nodes in both data centers are not able to > communicate with each other. When I do a nodetool status on a node in DC1, > the nodes in DC2 are stated as down. When tried in DC2, nodes in DC1 are > shown as down . > But in the cassandra logs, we can clearly see that handshaking is failing > every 5 seconds for communication between data centres. At TCP level, there > are too many fin_wait1 generated by cassandra which is still a puzzle . > Closed_wait top transitions due to this is very high. Due to this kind of > problem of TCP listen drops, we moved from 2.0.1 to 2.0.3. In 2.0.1, it was > within data center itself. But here it's between data centers. If it has > anything to do with the snitch configuration, I am using > GossipingPropertyFileSnitch. > This clearly started happening post private link failure. Any idea on this? > Cassandra version used is 2.0.3 -- This message was sent by Atlassian JIRA (v6.1.5#6160)