Further observations
I started suspecting that in the double fault scenario as below,
auto-down-unreachable-after
can play a significant role. As suspected, for TEST SCENARIO 3 below, when
auto-down-unreachable-after was enabled with 10s delay when nodes moved
from unreachable 'condition' (I understand that unreachable mainly a flag
in cluster-status and not a state by itself) to Removed State ,TEST
SCENARIO 3 below was able to allow the 3rd node join after all nodes 'saw'
3rd node as Down.
I very well understand the consequence of enabling auto-down-unreachable-after
config parameter. That can lead to an array of issues (including but not
limited to)
false-positives in case of network partitions
I request the clustering experts to comment if there are any alternatives
other than enabling auto-down-unreachable-after.
Thanks in advance
Regards
Muthu
On Monday, 4 December 2017 14:34:15 UTC+5:30, Muthukumaran Kothandaraman
wrote:
>
> Hi,
>
> I am trying to validate below statements in Akka Docs with slight twist in
> the scenario. After first seed node is brought down and before restarting
> the first seed-node, I try to restart another surviving node of cluster
>
> "*When a new node is started it sends a message to all seed nodes and
> then sends join command to the one that answers first.*"
>
> In conjunction with
>
> "*Once more than two seed nodes have been started it is no problem to
> shut down the first seed node. If the first seed node is restarted, it will
> first try to join the other seed nodes in the existing cluster.* "
>
> My configuration for 3 nodes (ie. 3 JVMs running on same machine) is as
> following in same order :
>
> cluster {
>
> // URL format is 'akka://' for artery. When netty is to be enabled,
> // change this to 'akka.tcp://'
> seed-nodes = [
> "akka.tcp://ClusterSystem@127.0.0.1:25510", // 1st Seed Node
> "akka.tcp://ClusterSystem@127.0.0.1:25520", // 2nd Seed Node
> "akka.tcp://ClusterSystem@127.0.0.1:25530" // 3rd Seed Node
> ]
>
> seed-node-timeout = 12s
>
> #auto-down-unreachable-after = 10s
>
> #allow-weakly-up-members = on
> }
>
>
>
> *TEST SCENARIO 1 (PASSED)* : Bring up nodes in any order and form the cluster
>
>
> *TEST SCENARIO 2 (PASSED)* : After cluster is formed, bring down the first
> seed node (ie. akka.tcp://ClusterSystem@127.0.0.1:25510).
>
> Ensure remaining cluster works without any disruption
>
>
> *TEST SCENARIO 3 (NOT PASSING**)* : Bring down 3rd node
> (akka.tcp://ClusterSystem@127.0.0.1:25530) when 1st seed node is still DOWN.
> And then bring up 3rd node.
>
> As I understand from first referred
> statement from documentation, 3rd seed node must be able to join 2nd
> seed-node even when 1st node is down
>
> But the observation is that 3rd seed node
> does NOT join the cluster.
>
>
> Of course, only when 2nd node is surviving and I bring up 1st as well as 3rd
> node, cluster forms correctly again
>
>
>
> *Clarification :*
>
>
> 1) Documentation mentions "once more than 2 seed nodes have been started".
> Does that mean that my TEST SCENARIO 3 above is invalid ? In other words, at
> any point in time,
>
> I should have at least 2 seed nodes of cluster to be alive make the TEST
> SCENARIO 3 above pass ?
>
>
> 2) And, if there are double faults like TEST SCENARIO 3, will cluster not
> converge till we perform full cluster reboot or bring back the first node in
> seed node list ?
>
>
>
> Regards
>
> Muthu
>
>
>
>
>
>
>
>
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
