[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025553#comment-14025553 ] Jason Brown commented on CASSANDRA-6523: +1 to @driftx's patch Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams Priority: Minor Fix For: 2.1.0 Attachments: 6523.txt New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14020303#comment-14020303 ] Chris Burroughs commented on CASSANDRA-6523: Jason, thanks for the suggestion. I think there are a very different ways I could work around it. At this point though I'd like to see it cleared up so others don't trip over it too. Tested driftx's patch on top of 2.0.8 with two DCs and the same NAT setup. * Started a node in DC A with auto_bootrap:false and it came up happy. * Started a second node in DC A, and it successfully bootstrapped. * Stopped all DC A nodes, started a node in DC B with auto_bootrap:true. It failed with Unable to contact any seeds!' * Started all DC A nodes, nodes in B could bootstrap. So it looks like it works from my point of view. Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams Priority: Minor Fix For: 2.1.0 Attachments: 6523.txt New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014663#comment-14014663 ] Juho Mäkinen commented on CASSANDRA-6523: - I have a similar issue described in CASSANDRA-7292 and in there I use the public ip addresses in the seeds list, but it does not help in my case. These two issues might or might not be connected. Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams Priority: Minor Fix For: 1.2.17, 2.0.9, 2.1 rc1 Attachments: 6523.txt New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014299#comment-14014299 ] Jason Brown commented on CASSANDRA-6523: [~cburroughs] Not sure how usable this suggestion is, but you could use the external addresses in the seeds list instead of the internal (or mixed per-dc). As GPFS is a 'reconnecting snitch', once the connection of the IP addr is established, the nodes in the same DC will switch over to the internal IP (listen_address). The length of time from node startup to switching the connection to internal IP should be rather short, so your traffic shouldn't be 'unusual' for much time. However, what happens during that time may be a deal breaker for using the external IPs everywhere in the seeds list. I bring this up as that's what we did when running multi-region in ec2, and is what EC2MRS facilitates. Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams Priority: Minor Fix For: 1.2.17, 2.0.9, 2.1 rc1 Attachments: 6523.txt New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007145#comment-14007145 ] Chris Burroughs commented on CASSANDRA-6523: I don't think that's an unreasonable requirement and I can roll with it but if there is an easy way the check could be made contact based instead of string equality based that would be nice. Starting a cluster is often the first thing someone will do with Cassandra and NATs, firewalls hijacking DNS requests, reconnecting snitches all make this complicated (Top results on stack overflow disagree for example [1] [2]). Barring a more clever check I suggest changing the message to something like Seed check failed! At least one boardcast_address must match IPs from seed list. Seeing Handshaking with foo from the seed list and then Unable to contact any seeds! is a confusing experience. [1] http://stackoverflow.com/questions/21261098/cassandra-unable-to-contact-seeds-if-using-aws-elastic-ip-address-only-works [2] http://stackoverflow.com/questions/20690987/apache-cassandra-unable-to-gossip-with-any-seeds Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14007252#comment-14007252 ] Brandon Williams commented on CASSANDRA-6523: - Part of the problem, is we can't know a seed's broadcast_address without, of course, them gossiping it to us. We could possibly extend the 'talked to a seed' check to loop through the endpoints we know about and look for a matching broadcast_address, though that's a bit expensive and is kind of cheating since we want to know we talked to a seed directly, not that we were told about a seed, but I don't think that actually matters. I'll have to think about it some. bq. Barring a more clever check I suggest changing the message to something like Seed check failed! At least one boardcast_address must match IPs from seed list. Well, that's not entirely accurate either, since if you aren't using BCA at all, it's confusing. We could print out the seeds with the error though, so at least between that and the handshaking messages you'd get a clue. Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams Priority: Minor New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005965#comment-14005965 ] Chris Burroughs commented on CASSANDRA-6523: Still present in 2.0.x series. There are a variety of stack overflow and mailing list threads threads with Unable to contact any seeds. I think the problem is that CASSANDRA-5768 isn't checking have I contacted a seed but am I connected to one of these IP addresses. That ends up being requirement that the seeds/broadcast/listen address line up in a particular way. Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006502#comment-14006502 ] Brandon Williams commented on CASSANDRA-6523: - It's comparing the actual IP address it received contact from to the seed list, so if it's being contacted by the broadcast address, but the seed is listed with the internal address, there won't be a match. So, you'd probably want a different seed list for DC 1 and DC 2, with the listen/broadcast switched for the seeds that aren't local. Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs Assignee: Brandon Williams New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13955303#comment-13955303 ] Chris Burroughs commented on CASSANDRA-6523: I believe it's a regression caused by CASSANDRA-5768 Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6523) Unable to contact any seeds! with multi-DC cluster and listen != broadcast address
[ https://issues.apache.org/jira/browse/CASSANDRA-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934138#comment-13934138 ] Jonathan Ellis commented on CASSANDRA-6523: --- Is this still a problem with 5768 fixed? Unable to contact any seeds! with multi-DC cluster and listen != broadcast address Key: CASSANDRA-6523 URL: https://issues.apache.org/jira/browse/CASSANDRA-6523 Project: Cassandra Issue Type: Bug Components: Core Environment: 1.2.13ish Reporter: Chris Burroughs New cluster: * Seeds: list of 6 internal IPs * listen address: internal ip * broadcast: external ip Two DC cluster, using GPFS where the external IPs are NATed. Clusters fails to start with Unable to contact any seeds! * Fail: Try to start a seed node * Fail: Try to start two seed nodes at the same time in the same DC * Success: Start two seed nodes at the same time in different DCs. Presumably related to CASSANDRA-5768 -- This message was sent by Atlassian JIRA (v6.2#6252)
