On 6/16/2011 9:36 PM, Dan Hendry wrote:
"Help me out here. I'm trying to visualize a situation where the
clients can access all the C* nodes but the nodes can't access each
other. I don't see how that can happen on a regular ethernet subnet
in one data center. Well, I"m sure there is a case that you can point
out. Ok, I will concede that this is an issue for some network
configurations."
First rule of designing/developing/operating distributed systems:
assume anything and everything can and will happen, regardless of
network configuration or hardware.
This specific situation actually HAS happened to me. Our Cassandra
nodes accept client connections on one ethernet interface on one
network (the production network) yet communicate with each other on a
separate ethernet interface on a separate network which is Cassandra
specific. This was done mainly due to the relatively large inter-node
Cassandra bandwidth requirements in comparison to client bandwidth
requirements. At one point, the switch for the cassandra network went
down so clients could connect yet the cassandra nodes could not talk
to eachother. (We write at ONE and read at ALL so everything behaved
as expected).
Funny, but that's the exact same setup I'm running. But, I'm not a
network guy and kind of assumed it wasn't so typical. Plus, lately I've
had my mind on a cloud setup.
