Looks like some problem with our monitoring framework. Thanks for you help !
On Mon, Apr 18, 2016 at 2:46 PM, Anishek Agarwal <anis...@gmail.com> wrote:
> OS used : Cent OS 6 on all nodes except *10*.125.138.59 ( which runs Cent
> OS 7)
> All of them are running Cassandra 2.0.17
>
> output of the test :
>
> host ip: 10.124.114.113
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.108
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.110
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.118
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.125.138.59
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.97
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.105
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
> host ip: 10.124.114.98
>
> host DC : WDC
>
> distance of host: LOCAL
>
> host is up: true
>
> cassandra version : 2.0.17
>
>
> On Fri, Apr 15, 2016 at 6:47 PM, Eric Stevens <migh...@gmail.com> wrote:
>
>> Thanks for that, that helps a lot. The next thing to check might be
>> whether or not your application actually has access to the other nodes.
>> With that topology, and assuming all the nodes you included in your
>> original graph are in the 'WDC' data center, I'd be inclined to look for a
>> network issue of some kind.
>>
>> Also, it probably doesn't matter, but what OS / Distribution are you
>> running the servers and clients on?
>>
>> Check with netcat or something that you can reach all the configured
>> ports from your application server, but also the driver itself offers some
>> introspection into its view of individual connection health. This is a
>> little bit ugly, but this is how we include information about connection
>> status in an API for health monitoring from a Scala application using the
>> Java driver; hopefully you can use it to see how to access information
>> about the driver's view of host health from the application's perspective.
>> Most importantly I'd suggest looking for host.isUp status and
>> LoadBalancingPolicy.distance(host) to see that it considers all the hosts
>> in your target datacenter to be LOCAL.
>>
>> "hosts" -> {
>> val hosts: Map[String, Map[String, mutable.Set[Host]]] =
>> connection.getMetadata
>> .getAllHosts.asScala
>> .groupBy(_.getDatacenter)
>> .mapValues(_.groupBy(_.getRack))
>> val lbp: LoadBalancingPolicy =
>> connection.getConfiguration.getPolicies.getLoadBalancingPolicy
>> JsObject(hosts.map { case (dc: String, rackAndHosts) =>
>> dc -> JsObject(rackAndHosts.map { case (rack: String, hosts:
>> mutable.Set[Host]) =>
>> rack -> JsArray(hosts.map { host =>
>> Json.obj(
>> "address" -> host.getAddress.toString,
>> "socketAddress" -> host.getSocketAddress.toString,
>> "cassandraVersion" -> host.getCassandraVersion.toString,
>> "isUp" -> host.isUp,
>> "hostDistance" -> lbp.distance(host).toString
>> )
>> }.toSeq)
>> }.toSeq)
>> }.toSeq)
>> },
>>
>>
>> On Wed, Apr 13, 2016 at 10:50 PM Anishek Agarwal <anis...@gmail.com>
>> wrote:
>>
>>> here is the output: every node in a single DC is in the same rack.
>>>
>>> Datacenter: WDC5
>>>
>>> ================
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> -- Address Load Tokens Owns (effective) Host ID
>>> Rack
>>>
>>> UN 10.125.138.33 299.22 GB 256 64.2%
>>> 8aaa6015-d444-4551-a3c5-3257536df476 RAC1
>>>
>>> UN 10.125.138.125 329.38 GB 256 70.3%
>>> 70be44a2-de17-41f1-9d3a-6a0be600eedf RAC1
>>>
>>> UN 10.125.138.129 305.11 GB 256 65.5%
>>> 0fbc7f44-7062-4996-9eba-2a05ae1a7032 RAC1
>>>
>>> Datacenter: WDC
>>>
>>> ===============
>>>
>>> Status=Up/Down
>>>
>>> |/ State=Normal/Leaving/Joining/Moving
>>>
>>> -- Address Load Tokens Owns (effective) Host ID
>>> Rack
>>>
>>> UN 10.124.114.105 151.09 GB 256 38.0%
>>> c432357d-bf81-4eef-98e1-664c178a3c23 RAC1
>>>
>>> UN 10.124.114.110 150.15 GB 256 36.9%
>>> 6f92d32e-1c64-4145-83d7-265c331ea408 RAC1
>>>
>>> UN 10.124.114.108 170.1 GB 256 41.3%
>>> 040ae7e5-3f1e-4874-8738-45edbf576e12 RAC1
>>>
>>> UN 10.124.114.98 165.34 GB 256 37.6%
>>> cdc69c7d-b9d6-4abd-9388-1cdcd35d946c RAC1
>>>
>>> UN 10.124.114.113 145.22 GB 256 35.7%
>>> 1557af04-e658-4751-b984-8e0cdc41376e RAC1
>>>
>>> UN 10.125.138.59 162.65 GB 256 38.6%
>>> 9ba1b7b6-5655-456e-b1a1-6f429750fc96 RAC1
>>>
>>> UN 10.124.114.97 164.03 GB 256 36.9%
>>> c918e497-498e-44c3-ab01-ab5cb4d48b09 RAC1
>>>
>>> UN 10.124.114.118 139.62 GB 256 35.1%
>>> 2bb0c265-a5d4-4cd4-8f50-13b5a9a891c9 RAC1
>>>
>>> On Thu, Apr 14, 2016 at 4:48 AM, Eric Stevens <migh...@gmail.com> wrote:
>>>
>>>> The output of nodetool status would really help answer some questions.
>>>> I take it the 8 hosts in your graph are in the same DC. Are the four
>>>> serving writes in the same logical or physical rack (as Cassandra sees it),
>>>> while the others are not?
>>>>
>>>> On Tue, Apr 12, 2016 at 10:48 PM Anishek Agarwal <anis...@gmail.com>
>>>> wrote:
>>>>
>>>>> We have two DC one with the above 8 nodes and other with 3 nodes.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 12, 2016 at 8:06 PM, Eric Stevens <migh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Maybe include nodetool status here? Are the four nodes serving reads
>>>>>> in one DC (local to your driver's config) while the others are in
>>>>>> another?
>>>>>>
>>>>>> On Tue, Apr 12, 2016, 1:01 AM Anishek Agarwal <anis...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> hello,
>>>>>>>
>>>>>>> we have 8 nodes in one cluster and attached is the traffic patterns
>>>>>>> across the nodes.
>>>>>>>
>>>>>>> its very surprising that only 4 nodes show transmitting (purple)
>>>>>>> packets.
>>>>>>>
>>>>>>> our driver configuration on clients has the following load balancing
>>>>>>> configuration :
>>>>>>>
>>>>>>> new TokenAwarePolicy(
>>>>>>> new
>>>>>>> DCAwareRoundRobinPolicy(configuration.get(Constants.LOCAL_DATA_CENTRE_NAME,
>>>>>>> "WDC")),
>>>>>>> true)
>>>>>>>
>>>>>>>
>>>>>>> any idea what is that we are missing which is leading to this skewed
>>>>>>> data read patterns
>>>>>>>
>>>>>>> cassandra drivers as below:
>>>>>>>
>>>>>>> <dependency>
>>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>>> <artifactId>cassandra-driver-core</artifactId>
>>>>>>> <version>2.1.6</version>
>>>>>>> </dependency>
>>>>>>> <dependency>
>>>>>>> <groupId>com.datastax.cassandra</groupId>
>>>>>>> <artifactId>cassandra-driver-mapping</artifactId>
>>>>>>> <version>2.1.6</version>
>>>>>>> </dependency>
>>>>>>>
>>>>>>> cassandra version is 2.0.17
>>>>>>>
>>>>>>> Thanks in advance for the help.
>>>>>>>
>>>>>>> Anishek
>>>>>>>
>>>>>>>
>>>>>
>>>
>
