I improved this the other week, but it has not been released yet. https://github.com/akka/akka/pull/24315
It would be great if you could take snapshot version 2.5-20180202-180000 for a spin and see if that works better for those tests. See https://doc.akka.io/docs/akka/current/project/links.html#snapshots-repository for info about how to add the snapshot repository to your build. Thanks, Patrik On Sun, Feb 4, 2018 at 11:56 PM, <lemms...@gmail.com> wrote: > Hi everyone, I have a question pertaining to the multi data center > behavior. In a nutshell, I'm wondering what the expected behavior is if > all of the nodes (especially the last one) in a data center go down, and > how the other data centers would respond to that. > > In my experimenting, when the last node goes down in a data center > (gracefully or not), and there are nodes remaining in other data centers, > they are unable to remove that node from the system. And if that node is > restarted, it cannot rejoin. I'm wondering if that is expected behavior or > if I've done something wrong. > > I've created a toy example that illustrates the behavior I'm encountering > (on the latest Akka release). > > Node "A" exists in data center "DC_A" > Node "B" exists in data center "DC_B" > Node "C" does not yet exist in data center "DC_B" (e.g. I haven't turned > it on yet) > > Auto-downing is off, but I have an API setup where I can tell any given > Node to tell any other Node to either Down or Leave the cluster. > > 1. B gracefully exits the cluster. > 2. DC_B is now empty > 3. A sees that B is in "Exiting" state. B is never actually > downed--it exists unto perpetuity. > 4. A keeps trying to reconnect to B, so I want to clean up its state. > But if I tell A to down B, B continues to exist in the Exiting state (as > far as A's view of the Cluster goes) > 5. If I start B up again, it cannot rejoin: "A" will spam its logs > with "New incarnation of existing member... is trying to join. Existing > member will be removed from the cluster and then new member will be allowed > to join." > 6. But the status quo continues. A keeps trying to reconnect to B, > but B exists forever in the Exiting state. I continue telling A to down B > manually (having verified that my code is correct for that), but nothing > happens. > 7. The only way I can figure out how to resolve the situation is to > start a new node in B's data center. So now, I start up C, which joins > DC_B as its only member. Eventually C figures out that B should be downed > and downs it, then re-allows it to enter the cluster. > > The above happens in the same way if B does not gracefully exit, e.g. I > give it a kill -9. > > The above works fine if I have A and B in the same data center. A marks B > as down and it leaves the cluster. > > I suppose my question is: is this expected behavior? I've read the multi > data center docs and can't get a handle on the expected behavior in that > situation. On one hand I understand that the point of the multi-datacenter > functionality is to keep them partitioned, so obviously A wouldn't do > anything automatically against B, or anything else outside of its data > center. But shouldn't I be able to tell nodes outside of a data center to > down all the nodes in another data center, seeing as I know they're > permanently down? Perhaps there is some other way of cleaning up the > cluster state that I'm not aware of. > > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Akka Tech Lead Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
