[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176925#comment-14176925 ] Joshua McKenzie commented on CASSANDRA-8084: +1 to v4 GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.12 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0-v4.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175219#comment-14175219 ] Joshua McKenzie commented on CASSANDRA-8084: Thoughts: * I don't like persisting both the peer and the connecting address all the way down the abstraction stack through StreamPlan to StreamSession. Both StreamPlan and StreamSession are incurring a pretty big burden just to be able to print the peer address alongside the private we're streaming to. * The name preferred used in StreamPlan implies that it will fall back to the from option if it can't hit preferred but that doesn't appear to be the case. Maybe a rename to 'connecting' in this context as well would be appropriate? * (nit) reseted should probably be resetted in OutboundTcpConnectionPool.java (predates this ticket, but while we're in the neighborhood...) Functionally looks sound to me. Only other thing I'd recommend is testing sstableloader as suggested previously. GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.12 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175223#comment-14175223 ] Yuki Morishita commented on CASSANDRA-8084: --- bq. I don't like persisting both the peer and the connecting address all the way down the abstraction stack That's what I wanted to avoid, but one reason we need both is that 'convic'ing from gossip does not work with private IPs, and I don't want to introduce system table lookup from inside streaming. I work on other renaming issues. GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.12 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175242#comment-14175242 ] Joshua McKenzie commented on CASSANDRA-8084: Fair point on the compromise. Better to persist that data locally than add the system table lookups into the streaming process and add an external dependency in that way. Consistency on naming should help with that part a bit. One last thing - it looks like there's some unused method signatures in StreamPlan we could take out and also normalize to (update testRequestEmpty() for instance) to help clear up some of the clutter and duplication in there. GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.12 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[
https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175588#comment-14175588
]
J.B. Langston commented on CASSANDRA-8084:
--
I don't think sstableloader is working right. Here is the output for
sstableloader itself:
{code}
automaton@ip-172-31-7-50:~/Keyspace1/Standard1$ sstableloader -d localhost `pwd`
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-320-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-326-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-325-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-283-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-267-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-211-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-301-Data.db
/home/automaton/Keyspace1/Standard1/Keyspace1-Standard1-jb-316-Data.db to
[/54.183.192.248, /54.215.139.161, /54.165.222.3, /54.172.118.222]
Streaming session ID: ac5dd440-5645-11e4-a813-3d13c3d3c540
progress: [/54.172.118.222 8/8 (100%)] [/54.183.192.248 8/8 (100%)]
[/54.165.222.3 8/8 (100%)] [/54.215.139.161 8/8 (100%)] [total: 100% -
2147483647MB/s (avg: 30MB/s)
{code}
Here is netstats on the node where it is running:
{code}
Responses n/a 0812
automaton@ip-172-31-7-50:~$ nodetool netstats
Mode: NORMAL
Bulk Load ac5dd440-5645-11e4-a813-3d13c3d3c540
/172.31.7.50 (using /54.183.192.248)
Receiving 8 files, 1059673728 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-10-Data.db
56468194/164372226 bytes(34%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-4-Data.db
27800/27800 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-3-Data.db
50674396/50674396 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-5-Data.db
68597334/68597334 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-7-Data.db
139068110/139068110 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-6-Data.db
12682638/12682638 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-9-Data.db
27800/27800 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-8-Data.db
68279024/68279024 bytes(100%) received from /172.31.7.50
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive Pending Completed
Commandsn/a 0 0
Responses n/a 0970
{code}
Here's netstats on the other node in the same DC:
{code}
automaton@ip-172-31-40-169:~$ nodetool netstats
Mode: NORMAL
Bulk Load ac5dd440-5645-11e4-a813-3d13c3d3c540
/172.31.7.50 (using /54.183.192.248)
Receiving 8 files, 1059673728 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-239-Data.db
68279024/68279024 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-245-Data.db
27800/27800 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-246-Data.db
43078602/50674396 bytes(85%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-240-Data.db
27800/27800 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-241-Data.db
12682638/12682638 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-243-Data.db
139068110/139068110 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-242-Data.db
164372226/164372226 bytes(100%) received from /172.31.7.50
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-244-Data.db
68597334/68597334 bytes(100%) received from /172.31.7.50
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive Pending Completed
Commandsn/a 0 249589
Responses
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175762#comment-14175762 ] Yuki Morishita commented on CASSANDRA-8084: --- bq. /172.31.7.50 (using /54.183.192.248) I think this is because 'localhost' resolved to /172.31.7.50 on sstableloader node. It is the same as before (except 'using ...' part) I think. But connection is actually made to broadcast address, it is showing 'using ...' part. sstableloader cannot determine whether nodes to stream are on the same dc or not. To do that, we have to provide topology manually, by file or by passing it through command line option. I can go further to solve the problem, but maybe in different JIRA? GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.12 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0-v4.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175804#comment-14175804 ] Jeremiah Jordan commented on CASSANDRA-8084: +1 to fixing sstable loader private ip stuff in a new JIRA. As long as it works normally (not using the private ips). GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.12 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0-v4.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[
https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173821#comment-14173821
]
J.B. Langston commented on CASSANDRA-8084:
--
Test v3; nodetool netstats looks good as well as actual ports used via netstat
-an. In the logs, I only see the internal IP mentioned in one place. Is this
the INFO line you were talking about?
{code}
INFO [STREAM-INIT-/172.31.5.143:43953] 2014-10-16 14:36:11,292
StreamResultFuture.java (line 121) [Stream
#c5fbdb90-5541-11e4-8eb3-c9fac3589773] Received streaming plan for Repair
INFO [STREAM-INIT-/172.31.5.143:43994] 2014-10-16 14:38:16,120
StreamResultFuture.java (line 121) [Stream
#10424ae0-5542-11e4-8eb3-c9fac3589773] Received streaming plan for Repair
{code}
GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE
clusters doesnt use the PRIVATE IPS for Intra-DC communications - When
running nodetool repair
-
Key: CASSANDRA-8084
URL: https://issues.apache.org/jira/browse/CASSANDRA-8084
Project: Cassandra
Issue Type: Bug
Components: Config
Environment: Tested this in GCE and AWS clusters. Created multi
region and multi dc cluster once in GCE and once in AWS and ran into the same
problem.
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS
NAME=Ubuntu
VERSION=12.04.3 LTS, Precise Pangolin
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=Ubuntu precise (12.04.3 LTS)
VERSION_ID=12.04
Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also
latest DSE version which is 4.5 and which corresponds to 2.0.8.39.
Reporter: Jana
Assignee: Yuki Morishita
Labels: features
Fix For: 2.0.11
Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0.txt
Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch )
used the PRIVATE IPS for communication between INTRA-DC nodes in my
multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran
nodetool repair -local. It works fine during regular reads.
Here are the various cluster flavors I tried and failed-
AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch +
(Prefer_local=true) in rackdc-properties file.
AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in
rackdc-properties file.
GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch +
(Prefer_local=true) in rackdc-properties file.
GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in
rackdc-properties file.
I am expecting with the above setup all of my nodes in a given DC all
communicate via private ips since the cloud providers dont charge us for
using the private ips and they charge for using public ips.
But they can use PUBLIC IPs for INTER-DC communications which is working as
expected.
Here is a snippet from my log files when I ran the nodetool repair -local -
Node responding to 'node running repair'
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254)
[repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree
to /54.172.118.222 for system_traces/sessions
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254)
[repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree
to /54.172.118.222 for system_traces/events
Node running repair -
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line
166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for
events from /54.172.118.222
Note: The IPs its communicating is all PUBLIC Ips and it should have used the
PRIVATE IPs starting with 172.x.x.x
YAML file values :
The listen address is set to: PRIVATE IP
The broadcast address is set to: PUBLIC IP
The SEEDs address is set to: PUBLIC IPs from both DCs
The SNITCHES tried: GPFS and EC2MultiRegionSnitch
RACK-DC: Had prefer_local set to true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173881#comment-14173881 ] Yuki Morishita commented on CASSANDRA-8084: --- That line has been in cassandra for a while. It is logged when a node receives streaming request. Displayed address is taken directly from socket so it displays actual connected address. New log looks like [Stream #-...] Starting streaming to /xxx.xxx.xxx.xxx through /yyy.yyy.yyy.yyy. where the latter address is private IP. GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.11 Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[
https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173927#comment-14173927
]
J.B. Langston commented on CASSANDRA-8084:
--
Confirmed the log messages:
{code}
INFO [StreamConnectionEstablisher:1] 2014-10-16 14:36:11,277
StreamSession.java (line 218) [Stream #c5fbdb90-5541-11e4-8eb3-c9fac3589773]
Starting streaming to /54.183.192.248 through /172.31.7.50
INFO [StreamConnectionEstablisher:2] 2014-10-16 14:38:16,083
StreamSession.java (line 218) [Stream #10424ae0-5542-11e4-8eb3-c9fac3589773]
Starting streaming to /54.183.192.248 through /172.31.7.50
INFO [StreamConnectionEstablisher:1] 2014-10-16 14:39:53,600
StreamSession.java (line 218) [Stream #4a9133f0-5542-11e4-8eb3-c9fac3589773]
Starting streaming to /54.183.192.248 through /172.31.7.50
INFO [StreamConnectionEstablisher:2] 2014-10-16 14:40:50,476
StreamSession.java (line 218) [Stream #6c5b4200-5542-11e4-8eb3-c9fac3589773]
Starting streaming to /54.183.192.248 through /172.31.7.50
{code}
Everything looks like it's working as expected. I haven't tested sstableloader
as suggested by [~jjordan].
GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE
clusters doesnt use the PRIVATE IPS for Intra-DC communications - When
running nodetool repair
-
Key: CASSANDRA-8084
URL: https://issues.apache.org/jira/browse/CASSANDRA-8084
Project: Cassandra
Issue Type: Bug
Components: Config
Environment: Tested this in GCE and AWS clusters. Created multi
region and multi dc cluster once in GCE and once in AWS and ran into the same
problem.
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS
NAME=Ubuntu
VERSION=12.04.3 LTS, Precise Pangolin
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=Ubuntu precise (12.04.3 LTS)
VERSION_ID=12.04
Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also
latest DSE version which is 4.5 and which corresponds to 2.0.8.39.
Reporter: Jana
Assignee: Yuki Morishita
Labels: features
Fix For: 2.0.11
Attachments: 8084-2.0-v2.txt, 8084-2.0-v3.txt, 8084-2.0.txt
Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch )
used the PRIVATE IPS for communication between INTRA-DC nodes in my
multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran
nodetool repair -local. It works fine during regular reads.
Here are the various cluster flavors I tried and failed-
AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch +
(Prefer_local=true) in rackdc-properties file.
AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in
rackdc-properties file.
GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch +
(Prefer_local=true) in rackdc-properties file.
GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in
rackdc-properties file.
I am expecting with the above setup all of my nodes in a given DC all
communicate via private ips since the cloud providers dont charge us for
using the private ips and they charge for using public ips.
But they can use PUBLIC IPs for INTER-DC communications which is working as
expected.
Here is a snippet from my log files when I ran the nodetool repair -local -
Node responding to 'node running repair'
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254)
[repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree
to /54.172.118.222 for system_traces/sessions
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254)
[repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree
to /54.172.118.222 for system_traces/events
Node running repair -
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line
166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for
events from /54.172.118.222
Note: The IPs its communicating is all PUBLIC Ips and it should have used the
PRIVATE IPs starting with 172.x.x.x
YAML file values :
The listen address is set to: PRIVATE IP
The broadcast address is set to: PUBLIC IP
The SEEDs address is set to: PUBLIC IPs from both DCs
The SNITCHES tried: GPFS and EC2MultiRegionSnitch
RACK-DC: Had prefer_local set to true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[
https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169468#comment-14169468
]
Yuki Morishita commented on CASSANDRA-8084:
---
bq. Is use of the broadcast address in netstats and the logs intentional?
Each node identifies other nodes by broadcast address, even messages are sent
through private IPs.
So currently logs and netstats show only broadcast IP.
I definitely can switch output for netstats to private IP when it is used. Or
show it alongside.
For logs, with some work it can be done. Though, I don't know if it helps a lot.
For example, repair log:
{noformat}
[repair #8cc731c0-52f4-11e4-916c-0800200c9a66] Endpoints /54.183.192.248 and
/54.215.139.161 are consistent for Standard1
{noformat}
is it better to show private IP for both nodes since they are communicating
through private IPs?
GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE
clusters doesnt use the PRIVATE IPS for Intra-DC communications - When
running nodetool repair
-
Key: CASSANDRA-8084
URL: https://issues.apache.org/jira/browse/CASSANDRA-8084
Project: Cassandra
Issue Type: Bug
Components: Config
Environment: Tested this in GCE and AWS clusters. Created multi
region and multi dc cluster once in GCE and once in AWS and ran into the same
problem.
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS
NAME=Ubuntu
VERSION=12.04.3 LTS, Precise Pangolin
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME=Ubuntu precise (12.04.3 LTS)
VERSION_ID=12.04
Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also
latest DSE version which is 4.5 and which corresponds to 2.0.8.39.
Reporter: Jana
Assignee: Yuki Morishita
Labels: features
Fix For: 2.0.11
Attachments: 8084-2.0.txt
Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch )
used the PRIVATE IPS for communication between INTRA-DC nodes in my
multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran
nodetool repair -local. It works fine during regular reads.
Here are the various cluster flavors I tried and failed-
AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch +
(Prefer_local=true) in rackdc-properties file.
AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in
rackdc-properties file.
GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch +
(Prefer_local=true) in rackdc-properties file.
GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in
rackdc-properties file.
I am expecting with the above setup all of my nodes in a given DC all
communicate via private ips since the cloud providers dont charge us for
using the private ips and they charge for using public ips.
But they can use PUBLIC IPs for INTER-DC communications which is working as
expected.
Here is a snippet from my log files when I ran the nodetool repair -local -
Node responding to 'node running repair'
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254)
[repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree
to /54.172.118.222 for system_traces/sessions
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254)
[repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree
to /54.172.118.222 for system_traces/events
Node running repair -
INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line
166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for
events from /54.172.118.222
Note: The IPs its communicating is all PUBLIC Ips and it should have used the
PRIVATE IPs starting with 172.x.x.x
YAML file values :
The listen address is set to: PRIVATE IP
The broadcast address is set to: PUBLIC IP
The SEEDs address is set to: PUBLIC IPs from both DCs
The SNITCHES tried: GPFS and EC2MultiRegionSnitch
RACK-DC: Had prefer_local set to true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[ https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169555#comment-14169555 ] J.B. Langston commented on CASSANDRA-8084: -- I think it is most important to show the private IP in netstats, and my vote would be to show both the public and private IP in that case. On the logs, I can see that would be more work to fix and I don't necessarily think it needs to show the private IP everywhere, but maybe on the messages that specifically concern streaming, we could show both. GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repair - Key: CASSANDRA-8084 URL: https://issues.apache.org/jira/browse/CASSANDRA-8084 Project: Cassandra Issue Type: Bug Components: Config Environment: Tested this in GCE and AWS clusters. Created multi region and multi dc cluster once in GCE and once in AWS and ran into the same problem. DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.3 LTS NAME=Ubuntu VERSION=12.04.3 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.3 LTS) VERSION_ID=12.04 Tried to install Apache Cassandra version ReleaseVersion: 2.0.10 and also latest DSE version which is 4.5 and which corresponds to 2.0.8.39. Reporter: Jana Assignee: Yuki Morishita Labels: features Fix For: 2.0.11 Attachments: 8084-2.0.txt Neither of these snitches(GossipFilePropertySnitch and EC2MultiRegionSnitch ) used the PRIVATE IPS for communication between INTRA-DC nodes in my multi-region multi-dc cluster in cloud(on both AWS and GCE) when I ran nodetool repair -local. It works fine during regular reads. Here are the various cluster flavors I tried and failed- AWS + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. AWS + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + GossipPropertyFileSnitch + (Prefer_local=true) in rackdc-properties file. GCE + Multi-REGION + Multi-DC + EC2MultiRegionSnitch + (Prefer_local=true) in rackdc-properties file. I am expecting with the above setup all of my nodes in a given DC all communicate via private ips since the cloud providers dont charge us for using the private ips and they charge for using public ips. But they can use PUBLIC IPs for INTER-DC communications which is working as expected. Here is a snippet from my log files when I ran the nodetool repair -local - Node responding to 'node running repair' INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,628 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/sessions INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,741 Validator.java (line 254) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Sending completed merkle tree to /54.172.118.222 for system_traces/events Node running repair - INFO [AntiEntropyStage:1] 2014-10-08 14:47:51,927 RepairSession.java (line 166) [repair #1439f290-4efa-11e4-bf3a-df845ecf54f8] Received merkle tree for events from /54.172.118.222 Note: The IPs its communicating is all PUBLIC Ips and it should have used the PRIVATE IPs starting with 172.x.x.x YAML file values : The listen address is set to: PRIVATE IP The broadcast address is set to: PUBLIC IP The SEEDs address is set to: PUBLIC IPs from both DCs The SNITCHES tried: GPFS and EC2MultiRegionSnitch RACK-DC: Had prefer_local set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[
https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166908#comment-14166908
]
J.B. Langston commented on CASSANDRA-8084:
--
I tested and it appears to work. Here is the cluster I am testing with:
{code}
Datacenter: DC1_EAST
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 54.165.222.3711.26 MB 1 25.0%
dd449706-2059-4b65-ae98-0012d2cf8f67 rack1
UN 54.172.118.222 561.14 MB 1 25.0%
18cd7d0a-74ca-4835-a7ff-7ffaa92b35ef rack1
Datacenter: DC1_WEST
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 54.183.192.248 721.2 MB 1 25.0%
c4dd37f1-d937-4876-8669-f0b01a3942db rack1
UN 54.215.139.161 909.26 MB 1 25.0%
16499349-8cef-4a62-a99c-ab145cb70921 rack1
I wasn't sure initially because the logs and `nodetool netstats` still show the
broadcast address. You can see here that nodetool netstats, when run on
54.215.139.161, shows we are streaming from 54.183.192.248 (the broadcast
address of the other node in the same DC):
{code}
Mode: NORMAL
Repair dbc7ea40-5082-11e4-8190-c9fac3589773
/54.183.192.248
Receiving 9 files, 229856794 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-100-Data.db
58878176/58878176 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-106-Data.db
97856/97856 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-109-Data.db
69407704/69407704 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-108-Data.db
3203116/3203116 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-102-Data.db
12545306/12545306 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-103-Data.db
69407704/69407704 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-104-Data.db
1536228/1536228 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-105-Data.db
12589230/12589230 bytes(100%) received from /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-107-Data.db
2191474/2191474 bytes(100%) received from /54.183.192.248
Sending 5 files, 109645980 bytes total
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-87-Data.db
14323672/14323672 bytes(100%) sent to /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-97-Data.db
20581730/20581730 bytes(100%) sent to /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-98-Data.db
3161694/3161694 bytes(100%) sent to /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-95-Data.db
69407704/69407704 bytes(100%) sent to /54.183.192.248
/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-jb-99-Data.db
2171180/2171180 bytes(100%) sent to /54.183.192.248
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool NameActive Pending Completed
Commandsn/a 01495191
Responses n/a 0 714928
{code}
However, the output of `sudo netstat -anp | grep 7000 | sort -k5` shows that we
are only connecting to the local node on its listen address (172.31.7.50):
{code}
tcp0 0 172.31.5.143:7000 0.0.0.0:* LISTEN
17279/java
tcp0 0 172.31.5.143:7000 172.31.5.143:34936 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:7000 172.31.5.143:34937 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:7000 172.31.5.143:34938 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:34936 172.31.5.143:7000 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:34937 172.31.5.143:7000 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:34938 172.31.5.143:7000 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:7000 172.31.7.50:52125 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:7000 172.31.7.50:52126 ESTABLISHED
17279/java
tcp0 0 172.31.5.143:57502 172.31.7.50:7000
[jira] [Commented] (CASSANDRA-8084) GossipFilePropertySnitch and EC2MultiRegionSnitch when used in AWS/GCE clusters doesnt use the PRIVATE IPS for Intra-DC communications - When running nodetool repai
[
https://issues.apache.org/jira/browse/CASSANDRA-8084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163680#comment-14163680
]
J.B. Langston commented on CASSANDRA-8084:
--
Here is the AWS cluster used to reproduce this:
{code}
automaton@ip-172-31-0-237:~$ nodetool status
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: aws_east
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.165.86.246 304.01 MB 256 26.8% 1042deb8-5395-42b1-adf4-2a373149b052 rack1
UN 54.209.121.225 302.82 MB 256 21.8% 7e7499c2-acfb-4eda-b786-7878907038b8 rack1
Datacenter: aws_west
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 54.183.246.79 79.01 MB 256 24.7% 9a4450a4-d00b-407c-8217-464ca5d3d74c rack1
UN 54.183.249.149 319.14 MB 256 26.7% cb6579d4-3eac-48c6-a8c0-ca30071a97e8 rack1
{code}
Here is the test case I ran to reproduce this:
1) Run cassandra-stress once to create Keyspace1 and Standard1 CF.
2) Alter keyspace with replication to all nodes:
{code}
ALTER KEYSPACE Keyspace1 WITH replication = {
'class':
'NetworkTopologyStrategy',
'aws_east': '2',
'aws_west': '2'
};
{code}
3) Shut down one of the nodes in aws_west.
4) Run cassandra-stress on the other node in aws-west (just cassandra-stress
with no options). Let it finish.
5) Start back up the node.
6) Run nodetool repair -local
7) Repair and streaming messages in system.log will show that it is using the
broadcast IP for nodes in the same DC. You can also watch the connections
being established over the broadcast IP with this command:
{code}
sudo netstat -anp | grep 7000 | sort -k5
{code}
This was conducted on DSE with GPFS. We should repeat with EC2MRS on DSE and
with GPFS on Apache Cassandra/DSC.
Here is the netstat output showing that it is establishing connections to the
node in the same DC (54.183.249.149). This command is being run on
54.183.246.79, so it should have used the private 172 address to talk to
54.183.249.149 instead.
{code}
automaton@ip-172-31-0-237:~$ sudo netstat -anp | grep 7000 | sort -k5
tcp 0 0 172.31.0.237:7000 0.0.0.0:* LISTEN 8959/java
tcp 0 0 172.31.0.237:7000 172.31.0.237:54148 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:7000 172.31.0.237:54149 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:7000 172.31.0.237:54150 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:54148 172.31.0.237:7000 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:54149 172.31.0.237:7000 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:54150 172.31.0.237:7000 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:7000 172.31.4.163:56894 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:7000 172.31.4.163:56895 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:55510 172.31.4.163:7000 ESTABLISHED 8959/java
tcp 0 35 172.31.0.237:55504 172.31.4.163:7000 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:7000 54.165.86.246:36101 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:50600 54.165.86.246:7000 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:50606 54.165.86.246:7000 ESTABLISHED 8959/java
tcp 1 0 172.31.0.237:60588 54.183.249.149:7000 CLOSE_WAIT 8959/java
tcp 0 0 172.31.0.237:60587 54.183.249.149:7000 ESTABLISHED 8959/java
tcp 0 0 172.31.0.237:60505 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60508 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60509 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60511 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60513 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60514 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60515 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60517 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60521 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60523 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60524 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60527 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60528 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60532 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60534 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60536 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60538 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60544 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60546 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60552 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60554 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60560 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60562 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60564 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60565 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60566 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60568 54.183.249.149:7000 TIME_WAIT -
tcp 0 0 172.31.0.237:60570 54.183.249.149:7000 TIME_WAIT -
tcp 0 0
