I have not looked at the logs but you find answer to your last question in http://doc.akka.io/docs/akka/2.4/scala/cluster-usage.html#How_To_Cleanup_when_Member_is_Removed
/Patrik fre 5 aug. 2016 kl. 22:31 skrev Eric Swenson <e...@swenson.org>: > One more clue as to the cluster daemon's shutting itself down. Earlier in > the logs (although prior to several successful requests being handled), I > find this: > > [INFO] [08/05/2016 05:04:45.042] > [ClusterSystem-akka.actor.default-dispatcher-5] > [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp:// > ClusterSystem@10.0.3.103:2552] - Leader can currently not perform its > duties, reachability status: [akka.tcp://ClusterSystem@10.0.3.103:2552 -> > akka.tcp://ClusterSystem@10.0.3.102:2552: Unreachable [Unreachable] (16), > akka.tcp://ClusterSystem@10.0.3.103:2552 -> akka.tcp:// > ClusterSystem@10.0.3.104:2552: Unreachable [Unreachable] (17), akka.tcp:// > ClusterSystem@10.0.3.103:2552 -> akka.tcp://ClusterSystem@10.0.3.176:2552: > Unreachable [Unreachable] (18), akka.tcp://ClusterSystem@10.0.3.103:2552 > -> akka.tcp://ClusterSystem@10.0.3.240:2552: Unreachable [Unreachable] > (19)], member status: [akka.tcp://ClusterSystem@10.0.3.102:2552 Up > seen=false, akka.tcp://ClusterSystem@10.0.3.103:2552 Up seen=true, > akka.tcp://ClusterSystem@10.0.3.104:2552 Up seen=false, akka.tcp:// > ClusterSystem@10.0.3.176:2552 Up seen=false, akka.tcp:// > ClusterSystem@10.0.3.240:2552 Up seen=false] > > > All these log messages are from the node at IP address 10.0.3.103. So I'm > assuming this means the Leader is THIS node. It seems to be saying that it > cannot reach all the other cluster members, and because of that, it cannot > do its job. This probably accounts for why it decided to shut itself down. > > > There were 6 AWS EC2 instances running this application at the time (not > 10, as I said in an earlier message). However, the cluster membership > above, only shows 5 members at the time of this log message. Not sure what > happened to the other one. > > > [akka.tcp://ClusterSystem@10.0.3.102:2552 Up seen=false, > > akka.tcp://ClusterSystem@10.0.3.103:2552 Up seen=true, > > akka.tcp://ClusterSystem@10.0.3.104:2552 Up seen=false, > > akka.tcp://ClusterSystem@10.0.3.176:2552 Up seen=false, > > akka.tcp://ClusterSystem@10.0.3.240:2552 Up seen=false] > > > I'm going to assume, not having any other evidence, that AWS/EC2 > experienced some network issue at the time in question, and consequently > this node was not able to talk to the rest of the cluster and therefore > this member (the leader) shut down. I only have logs for one of the other > 5 cluster nodes, so I will check to see what that other node thought about > all this at the time. But I'm not very comfortable with the robustness of > akka here. I would have thought that the other cluster members could have, > perhaps, noticing that the Leader was unreachable (assuming they couldn't > reach it), and because I had auto-down-unreachable-after set (yes, yes, > I've sense replaced this with manual downing logic -- but that is on our > dev deployment and this issue happened on our staging deployment), elected > a new leader and carried on -- even if this node became catatonic. > > > This raises another point: When the ClusterDaemon shuts itself down, it > would appear that I should handle some event here (not sure how to do > that), to cause the entire JVM to terminate. This would cause AWS/ECS to > launch a new instance to join the remaining cluster. > > > Thoughts? -- Eric > > > > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
