[ https://issues.apache.org/jira/browse/CASSANDRA-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915386#comment-13915386 ]
Ananthkumar K S commented on CASSANDRA-6772: -------------------------------------------- [~brandon.williams] Agreed. It that's so, can someone let me know if it's a bug. Moreover, a firewall problem won't let the TCP connections happen between the two nodes. But here, as I mentioned, cassandra was retrying at the network layer and it was visible in netstat in both the server. We cannot replicate such a scenario as we have 60 other applications running on the same private link. So, as an use case, it should be a normal scenario in cassandra in detect and establish the connection once the connection comes up. When I reported a similar kind of a problem , an infinite loop was introduced to nullify these kind I race conditions. But it doesn't solve the problem but creates more load on TCP. Can you please review that part for such a scenario? > Cassandra inter data center communication broken > ------------------------------------------------ > > Key: CASSANDRA-6772 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6772 > Project: Cassandra > Issue Type: Bug > Environment: CentOS 6.0 > Reporter: Ananthkumar K S > Priority: Blocker > > I have two data enters DC1 and DC2. Both communicate via a private link. > Yesterday, we had a problem with a private link for 10 mins. From the time > the problem was resolved, nodes in both data centers are not able to > communicate with each other. When I do a nodetool status on a node in DC1, > the nodes in DC2 are stated as down. When tried in DC2, nodes in DC1 are > shown as down . > But in the cassandra logs, we can clearly see that handshaking is failing > every 5 seconds for communication between data centres. At TCP level, there > are too many fin_wait1 generated by cassandra which is still a puzzle . > Closed_wait top transitions due to this is very high. Due to this kind of > problem of TCP listen drops, we moved from 2.0.1 to 2.0.3. In 2.0.1, it was > within data center itself. But here it's between data centers. If it has > anything to do with the snitch configuration, I am using > GossipingPropertyFileSnitch. > This clearly started happening post private link failure. Any idea on this? > Cassandra version used is 2.0.3 -- This message was sent by Atlassian JIRA (v6.1.5#6160)