Thanks, Shasha.
It looks like there are two ways to make Cassandra work across regions,
changing the source code or using software like Vyatta and VIPs.
++ Changing the source code ++
Milind documents how to change the source code in 0.7.4 but warns that there
are significant limitations to this, which are not really discussed in his
paper. I think some people ran into a problem running nodetool repair after
this change, but it worked fine for others. Also, people worried about
inter-node communication not being encrypted last month, but with the
release of 0.8 this isn't a problem. Are there any other problems we can run
into by changing the source code this way?
How to change source code in 0.7.4:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/attachment/6191521/0/CassandraInstancesindifferentregions--EC2%5B1%5D.pdf?by-user=t
+++ Configuring VIPs +++
Shasha documented a method of using Vyatta (vyatta.com) and VIPs to
configure Cassandra across regions. This technique configures a VIP
interface (tunXXX) for each node and sets the listen address as the VIP, so
all inter-node communication occurs over the VPN tunnel. The RPC address is
set to 0.0.0.0 so it listens on all interfaces and does not use the VPN
tunnel.
- This approach seems to be better, but does anyone have instructions for
doing this in EC2 without Vyatta, which is paid software and costs around
$700? We're using Ubuntu 11.04 on EC2 and would prefer to use an open
source, free software vendor.
- Sameer
On Thu, May 5, 2011 at 10:48 PM, Sasha Dolgy sdo...@gmail.com wrote:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Re-IP-address-resolution-in-MultiDC-setup-EC2-VIP-td6306635.html
On May 6, 2011 3:07 AM, Sameer Farooqui cassandral...@gmail.com wrote:
We're trying to set up a Cassandra 0.8.0beta1 cluster across Amazon East
West regions. It does not work out of the box with the binaries and the
nodes in different regions end up setting this own clusters.
The problem is with Cassandra's Listening Address as described by Rui:
Using external IP causes Binding exception, because Cassandra, and java
socket, cannot listen on the external IP, which does not belong to this
node. Using private IP prevents nodes in other region from connecting
back
to this node with the private IP, which has been sent to other nodes for
cluster establishing.
In Feb 2011, Peter Fales posted: ... the trick is that you have must use
the external addresses in your storage-conf.xml, but since you don't
have
a NIC that can actually bind to those addresses, you need to listen on
the
internal addresses (or, more simply, all interfaces). At the time, I
was
not able to get the cross-region cluster to work without making changes
to
the Cassandra code. Perhaps things have evolved so that there are other
ways
to do it now.
Does anyone know if Cassandra 0.8 can be configured to run in multiple
Amazon regions without modifying the source code? And if not, has anybody
successfully done this with the 0.8 source code and shared the technique?
After some research we found that other people have also ran into this
issue
with versions 0.6.x and 0.7.x of Cassandra:
Fix for 0.6.5 by Peter Fales in Sept 2010:
http://www.mail-archive.com/dev@cassandra.apache.org/msg01832.html
http://www.mail-archive.com/user@cassandra.apache.org/msg05692.html
Fix for 0.7 by Rui:
http://rliu09.wordpress.com/2011/04/23/cassandra-across-ec2-regions/
- Sameer
Accenture Technology Labs