Let’s stick to Akka terminology as otherwise it gets confusing :-) Simply put: there is no re-join. You simply never DOWN, the node will “come back (from UNREACHABLE)” state, and things will just work - when using only AP features such as DData.
Yeah, perhaps we could spell out that certain features value C or A over the other, as indeed they are designed specifically to favour one more over the other, I’ll ticketify this feedback - thanks :) — Konrad `kto.so` Malawski Akka <http://akka.io> @ Lightbend <http://lightbend.com> On 1 September 2017 at 01:42:48, Igor Baltiyskiy (ios...@gmail.com) wrote: Thanks, Konrad, that clears things up for me! I indeed somehow missed the idea that a downed node never comes back. Perhaps the docs on auto-downing could stress this point. > Thus thinking in the same terms as a database node “re-joining” its cluster, about an Akka Cluster is not a good idea - a database can often more easily reconsile data of a “stray” node, because holding data is the only thing they do. Well, looks like Distributed Data should be able to re-join nicely, right? That was my point --- if I designed the application with re-joining in mind, and avoiding features that break during re-joining, then I'd be fine. So, roughly speaking, you might separate Akka features into the "CP" and "AP" camp, with most things (Akka Cluster included) historically in the "CP" one. Thanks Igor On Thursday, August 31, 2017 at 12:38:05 PM UTC+3, Konrad Malawski wrote: > > Igor, that’s basically the thing: > > - “They merge” if you *don’t* DOWN, but things are stay in UNREACHABLE > then, which could be fine in your use case. > - If you DOWN, there’s no way back. You said you never want to talk to it > again, and we’re guaranteeing that. > > Docs on the state transitions are here: http://doc.akka.io/docs/ > akka/current/scala/common/cluster.html > > This is not because we can’t implement leader election, but because it > would be against what this state was designed for – consistency. > > The Akka Cluster is designed to guarantee consistency once it has DOWNed > nodes, otherwise yeah, some node would “come back”, but it has been doing > stuff all the time, > and unlike a database, it could have been *anything*. Thus thinking in the > same terms as a database node “re-joining” its cluster, about an Akka > Cluster is not a good idea - > a database can often more easily reconsile data of a “stray” node, because > holding data is the only thing they do. > > (Even databases (cassandra) “down completely” nodes after some time of > unreachability and will not attempt to treat them as the same node if they > ever come back - these things happen over inactivity over multiple days, > and not the types of failures you talk about). > > — > Konrad `kto.so` Malawski > Akka <http://akka.io> @ Lightbend <http://lightbend.com> > > On 31 August 2017 at 18:24:51, Igor Baltiyskiy (ios...@gmail.com) wrote: > > > given enough time any cluster of size N with auto-down will eventually > end up as N single-node clusters. It is thus not a sustainable strategy > unless you invest the manpower to constantly manually rebuild the cluster. > > Do you mean that after the netsplit heals, clusters won't merge on their > own? > > Theoretically, they could: once a leader node sees another node that > thinks that it is a leader, they elect a leader among themselves, and nodes > from the loser cluster leave that cluster and join the winner cluster. It's > then up to the individual features to be able to cope with that. For > instance, I'd expect Distributed Data to able to cope with that, since the > ability merge is the central idea of CRDT. On the other hand, Akka > Persistence won't be able to cope with that. > > So the way I would use Cluster in this scenario is by not using features > that are not "AP" (Akka Persistence, probably Akka Singleton --- but see > question 3 in the original post), and instead using those that are --- > Distributed Data, cluster-aware routing. The idea is that netsplits do > happen, but they are transient, so most of the time the cluster operates as > a single whole; during netsplit, it can be broken into several clusters, > each of which will operate as before, but will need to merge once they > rejoin. > > Of course, if clusters do not "merge" automatically, then it's a no-go. So > --- could you confirm that it is the case? > > Regards > Igor > > On Wednesday, August 30, 2017 at 9:33:16 PM UTC+3, rkuhn wrote: >> >> Another way of putting this is that given enough time any cluster of size >> N with auto-down will eventually end up as N single-node clusters. It is >> thus not a sustainable strategy unless you invest the manpower to >> constantly manually rebuild the cluster. >> >> Regards, Roland >> >> Sent from my iPhone >> >> On 30. Aug 2017, at 18:25, Justin du coeur <jduc...@gmail.com> wrote: >> >> On Wed, Aug 30, 2017 at 11:24 AM, Igor Baltiyskiy <ios...@gmail.com> >> wrote: >> >>> I'd like to clarify the warning given in the documentation: >>> >>> > We recommend against using the auto-down feature of Akka Cluster in >>> production. This is crucial for correct behavior if you use Cluster >>> Singleton >>> <http://doc.akka.io/docs/akka/2.5/scala/cluster-singleton.html> or Cluster >>> Sharding <http://doc.akka.io/docs/akka/2.5/scala/cluster-sharding.html>, >>> especially together with Akka Persistence >>> <http://doc.akka.io/docs/akka/2.5/scala/persistence.html>. For Akka >>> Persistence with Cluster Sharding it can result in corrupt data in case of >>> network partitions. >>> (http://doc.akka.io/docs/akka/2.5/scala/cluster-usage.html# >>> auto-downing-do-not-use-) >>> >>> This passage raises several questions: >>> >>> 1. Does it mean that auto-downing is *always* a bad idea? Or is it only >>> a bad idea when using Cluster Singleton and Cluster Sharding, so that if >>> I'm not using them, using auto-downing in production is perfectly fine? >>> >> >> It's a more or less *guaranteed* disaster in the long run with Cluster >> Sharding; it's only a *likely* disaster without, depending on what your app >> is doing. But personally, I think it should never, ever be used. >> >> Look at it this way: the problem with auto-down is that it is likely, in >> the long run, that you will eventually wind up in a Split Brain situation, >> with two or more disconnected clusters, each of which thinks that it is The >> One True Cluster and Source of Truth. This will usually happen in a >> realistic network, due to the vagaries of network traffic and the >> fundamental dumbness of the auto-down algorithm. >> >> Now, ask yourself: if this is acceptable -- if it is okay to sometimes >> *not* be a Cluster -- why are you using Clustering at all? >> >> Basically, if your nodes need to be able to share information in a >> consistent way, you need Clustering, which means that Split-Brain is >> deadly, which means that you must not use auto-down. If the nodes *don't* >> need to be sharing information, you probably shouldn't be bothering with >> Clustering at all, so auto-down is irrelevant. >> >> That's the heart of the argument, and why I personally think auto-down >> should be simply excised from the library entirely: it is *never* safe to >> use in anything more than a toy environment, and its existence just >> confuses people... >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >> current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+...@googlegroups.com. >> To post to this group, send email to akka...@googlegroups.com. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+...@googlegroups.com. > To post to this group, send email to akka...@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
