Re: Cassandra UNREACHABLE node

2017-07-18 Thread Ben Slater 
I’m not sure it’s going to help you a lot as the issue is still open and
there is not a lot of info in the ticket but your issue looks a lot like
this JIRA to me:
https://issues.apache.org/jira/browse/CASSANDRA-13363

Cheers
Ben

On Tue, 18 Jul 2017 at 17:46 Shashikant Kulkarni (शशिकांत कुलकर्णी) <
shashikant.kulka...@gmail.com> wrote:

> Hello All,
>
> I have Cassandra v3.9, with 3 nodes in cluster. Replication factor 2
>
> 10.0.0.11,10.0.0.12,10.0.0.13
>
> 10.0.0.11,10.0.0.12 are seed nodes
>
> What could be the possible reasons for following error in
> /etc/cassandra/conf/debug.log
>
> The error is
>
> DEBUG [RMI TCP Connection(174)-127.0.0.1] 2017-07-12 04:47:49,002
> StorageProxy.java:2254 - Hosts not in agreement. Didn’t get a response from
> everybody: 10.0.0.13
>
> At the time of the error here are some statistics from all the servers
>
> [user1@ip-10-0-0-11 ~]$ nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address Load   Tokens   Owns (effective)  Host ID
>   Rack
> UN  10.0.0.12  2.55 MiB   256  67.2%
> 83a68750-2238-4a6e-87be-03a3d7246824  rack1
> UN  10.0.0.11  1.78 GiB   256  70.6%
> 052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
> UN  10.0.0.13  1.78 GiB   256  62.2%
> 86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1
>
> [user1@ip-10-0-0-11 ~]$ nodetool describecluster
> Cluster Information:
> Name: PiedmontCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.11]
>
> UNREACHABLE: [10.0.0.13]
>
>
> [user1@ip-10-0-0-13 ~]$ nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address Load   Tokens   Owns (effective)  Host ID
>   Rack
> UN  10.0.0.12  2.55 MiB   256  67.2%
> 83a68750-2238-4a6e-87be-03a3d7246824  rack1
> UN  10.0.0.11  1.78 GiB   256  70.6%
> 052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
> UN  10.0.0.13  1.78 GiB   256  62.2%
> 86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1
>
> [user1@ip-10-0-0-13 ~]$ nodetool describecluster
> Cluster Information:
> Name: PiedmontCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.13]
>
> UNREACHABLE: [10.0.0.11]
>
>
> [user1@ip-10-0-0-12 ~]$ nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address Load   Tokens   Owns (effective)  Host ID
>   Rack
> UN  10.0.0.12  2.55 MiB   256  67.2%
> 83a68750-2238-4a6e-87be-03a3d7246824  rack1
> UN  10.0.0.11  1.78 GiB   256  70.6%
> 052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
> UN  10.0.0.13  1.78 GiB   256  62.2%
> 86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1
>
> [user1@ip-10-0-0-12 ~]$ nodetool describecluster
> Cluster Information:
> Name: PiedmontCluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12,
> 10.0.0.11, 10.0.0.13]
> The above mentioned error is in the /var/log/cassandra/debug.log on
> 10.0.0.11
>
> Error in /var/log/cassandra/debug.php on 10.0.0.13 is
>
> DEBUG [RMI TCP Connection(4)-127.0.0.1] 2017-07-13 02:31:23,846
> StorageProxy.java:2254 - Hosts not in agreement. Didn’t get a response from
> everybody: 10.0.0.11
> ERROR [MessagingService-Incoming-/10.0.0.11] 2017-07-13 02:35:04,982
> CassandraDaemon.java:226 - Exception in thread
> Thread[MessagingService-Incoming-/10.0.0.11,5,main]
> java.lang.ArrayIndexOutOfBoundsException: 4
> at
>  
> org.apache.cassandra.db.filter.AbstractClusteringIndexFilter$FilterSerializer.deserialize(AbstractClusteringIndexFilter.java:74)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Deserializer.deserialize(SinglePartitionReadCommand.java:1041)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:696)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:626)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at org.

Cassandra UNREACHABLE node

2017-07-18 Thread शशिकांत कुलकर्णी 
Hello All,

I have Cassandra v3.9, with 3 nodes in cluster. Replication factor 2

10.0.0.11,10.0.0.12,10.0.0.13

10.0.0.11,10.0.0.12 are seed nodes

What could be the possible reasons for following error in
/etc/cassandra/conf/debug.log

The error is

DEBUG [RMI TCP Connection(174)-127.0.0.1] 2017-07-12 04:47:49,002
StorageProxy.java:2254 - Hosts not in agreement. Didn’t get a response from
everybody: 10.0.0.13

At the time of the error here are some statistics from all the servers

[user1@ip-10-0-0-11 ~]$ nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address Load   Tokens   Owns (effective)  Host ID
Rack
UN  10.0.0.12  2.55 MiB   256  67.2%
83a68750-2238-4a6e-87be-03a3d7246824  rack1
UN  10.0.0.11  1.78 GiB   256  70.6%
052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
UN  10.0.0.13  1.78 GiB   256  62.2%
86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1

[user1@ip-10-0-0-11 ~]$ nodetool describecluster
Cluster Information:
Name: PiedmontCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.11]

UNREACHABLE: [10.0.0.13]


[user1@ip-10-0-0-13 ~]$ nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address Load   Tokens   Owns (effective)  Host ID
Rack
UN  10.0.0.12  2.55 MiB   256  67.2%
83a68750-2238-4a6e-87be-03a3d7246824  rack1
UN  10.0.0.11  1.78 GiB   256  70.6%
052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
UN  10.0.0.13  1.78 GiB   256  62.2%
86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1

[user1@ip-10-0-0-13 ~]$ nodetool describecluster
Cluster Information:
Name: PiedmontCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12, 10.0.0.13]

UNREACHABLE: [10.0.0.11]


[user1@ip-10-0-0-12 ~]$ nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address Load   Tokens   Owns (effective)  Host ID
Rack
UN  10.0.0.12  2.55 MiB   256  67.2%
83a68750-2238-4a6e-87be-03a3d7246824  rack1
UN  10.0.0.11  1.78 GiB   256  70.6%
052fda9d-0474-4dfb-b2f8-0c5cbec15266  rack1
UN  10.0.0.13  1.78 GiB   256  62.2%
86438dc9-77e0-43b2-a672-5b2e7cf216bf  rack1

[user1@ip-10-0-0-12 ~]$ nodetool describecluster
Cluster Information:
Name: PiedmontCluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
3c8d9e82-c688-3d16-a3e9-b84894168283: [10.0.0.12,
10.0.0.11, 10.0.0.13]
The above mentioned error is in the /var/log/cassandra/debug.log on
10.0.0.11

Error in /var/log/cassandra/debug.php on 10.0.0.13 is

DEBUG [RMI TCP Connection(4)-127.0.0.1] 2017-07-13 02:31:23,846
StorageProxy.java:2254 - Hosts not in agreement. Didn’t get a response from
everybody: 10.0.0.11
ERROR [MessagingService-Incoming-/10.0.0.11] 2017-07-13 02:35:04,982
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.0.0.11,5,main]
java.lang.ArrayIndexOutOfBoundsException: 4
at
 
org.apache.cassandra.db.filter.AbstractClusteringIndexFilter$FilterSerializer.deserialize(AbstractClusteringIndexFilter.java:74)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.db.SinglePartitionReadCommand$Deserializer.deserialize(SinglePartitionReadCommand.java:1041)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:696)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.db.ReadCommand$Serializer.deserialize(ReadCommand.java:626)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.io.ForwardingVersionedSerializer.deserialize(ForwardingVersionedSerializer.java:50)
~[apache-cassandra-3.9.0.jar:3.9.0]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:114)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:190)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178)
~[apache-cassandra-3.9.0.jar:3.9.0]
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92)
~[apache-cassandra-3.9.0.jar:3.9.0]
No error in /var/log/cassandra/debug.php on 10.0.0.12

Remembe