Re: Traffic inconsistent across nodes

2016-04-12 Thread Eric Stevens
Maybe include nodetool status here? Are the four nodes serving reads in one DC (local to your driver's config) while the others are in another? On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal wrote: > hello, > > we have 8 nodes in one cluster and attached is the traffic patterns across > the node

Re: Traffic inconsistent across nodes

2016-04-12 Thread Anishek Agarwal
We have two DC one with the above 8 nodes and other with 3 nodes. On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens wrote: > Maybe include nodetool status here? Are the four nodes serving reads in > one DC (local to your driver's config) while the others are in another? > > On Tue, Apr 12, 2016,

Re: Traffic inconsistent across nodes

2016-04-13 Thread Eric Stevens
The output of nodetool status would really help answer some questions. I take it the 8 hosts in your graph are in the same DC. Are the four serving writes in the same logical or physical rack (as Cassandra sees it), while the others are not? On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal wrot

Re: Traffic inconsistent across nodes

2016-04-13 Thread Anishek Agarwal
here is the output: every node in a single DC is in the same rack. Datacenter: WDC5 Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.125.138.33 299.22 GB 256 64.2% 8a

Re: Traffic inconsistent across nodes

2016-04-15 Thread Eric Stevens
Thanks for that, that helps a lot. The next thing to check might be whether or not your application actually has access to the other nodes. With that topology, and assuming all the nodes you included in your original graph are in the 'WDC' data center, I'd be inclined to look for a network issue o

Re: Traffic inconsistent across nodes

2016-04-18 Thread Anishek Agarwal
OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent OS 7) All of them are running Cassandra 2.0.17 output of the test : host ip: 10.124.114.113 host DC : WDC distance of host: LOCAL host is up: true cassandra version : 2.0.17 host ip: 10.124.114.108 host DC : WDC dist

Re: Traffic inconsistent across nodes

2016-04-18 Thread Anishek Agarwal
Looks like some problem with our monitoring framework. Thanks for you help ! On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal wrote: > OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent > OS 7) > All of them are running Cassandra 2.0.17 > > output of the test : > > host ip

Re: Traffic inconsistent across nodes

2016-04-19 Thread Eric Stevens
Glad that worked out, that was going to be my next suspicion, since everything thought it was up and happy, I can't think of a way that Cassandra and the driver could both consider the cluster happy if some nodes were not transmitting at least some data (they have to at least for gossip). On Mon,