I think I understand what is going on and what we can consider to improve. The heartbeat messages don't include the system uid, and there fore the restarted system starts responding to heartbeat messages that are targeted to the old incarnation. Then the cluster marks it as reachable again, before the auto-down takes affect, i.e. it is never removed from the cluster. The new system tries to join, but that is not possible because the cluster already contains same host:port.
I think this is best solved by including system uid in the heartbeat messages, but that increase the payload size of these messages. An issue ticket would be good. Regards, Patrik On Wed, Nov 5, 2014 at 10:00 AM, Björn Antonsson < bjorn.antons...@typesafe.com> wrote: > Hi Behrad, > > On 5 November 2014 at 09:53:00, Behrad (behr...@gmail.com) wrote: > > > > 2014-11-05 11:59 GMT+03:30 Björn Antonsson <bjorn.antons...@typesafe.com>: > >> Hi Richard, >> >> On 5 November 2014 at 00:22:55, richard (harold.richard.sea...@gmail.com) >> wrote: >> >> I am seeing something similar with this github >> <https://github.com/searler/akka-datareplication-experimentation> code, >> based on akka-datareplication, using Akka 2.3.6 >> (That might be a little too complex for a ticket) >> >> Note that *auto-down-unreachable-after* is commented out >> >> >> If the old node is never downed and removed from the cluster, then the >> new node can never join. >> > > > > Does this mean we should always set auto-down to a small value so that we > can recover from > (and reconnect) > cluster > > note crashes? What is the "unreachable" -> "reachable state" state change > for then !? I'd expect that my node went to unreachable state again is > reachable when it's again up in between the failure detection threshold. > > It also isn't happening for me, in both cases. > > > If you want to have the nodes automatically be downed is a different issue > than the reachability. The states reachabel/unreachable is for a node > instance that experiences connection failures (network outages et.c.) but > not restarts, while the downing is necessary when a new node with the same > address/port as the old one is joining (in effect a restarted actor system). > > B/ > > > > > > >> B/ >> >> >> Started two instances, one on 2551 (the seed) and another on 1234. >> Enter text into each instance, which is correctly replicated to each. >> Kill and restart the 1234 instance. >> >> The new 1234 instance receives the current state (from 2551) and >> continues to >> replicate in both directions! >> >> The log on 2551 does indicate a problem >> [INFO] [11/04/2014 17:20:07.309] >> [ClusterSystem-akka.actor.default-dispatcher-20] >> [Cluster(akka://ClusterSystem)] Cluster Node >> [akka.tcp://ClusterSystem@localhost:2551] - Existing member >> [UniqueAddress(akka.tcp://ClusterSystem@localhost:1234,1772853420)] is >> trying to join, ignoring >> [INFO] [11/04/2014 17:20:17.319] >> [ClusterSystem-akka.actor.default-dispatcher-17] >> [Cluster(akka://ClusterSystem)] Cluster Node >> [akka.tcp://ClusterSystem@localhost:2551] - Existing member >> [UniqueAddress(akka.tcp://ClusterSystem@localhost:1234,1772853420)] is >> trying to join, ignoring >> [INFO] [11/04/2014 17:20:28.310] >> [ClusterSystem-akka.actor.default-dispatcher-3] >> [Cluster(akka://ClusterSystem)] Cluster Node >> [akka.tcp://ClusterSystem@localhost:2551] - Existing member >> [UniqueAddress(akka.tcp://ClusterSystem@localhost:1234,1772853420)] is >> trying to join, ignoring >> >> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> Björn Antonsson >> Typesafe <http://typesafe.com/> – Reactive Apps on the JVM >> twitter: @bantonsson <http://twitter.com/#!/bantonsson> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "Akka User List" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/akka-user/AdRSv2yuwo4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> akka-user+unsubscr...@googlegroups.com. >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > --Behrad > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > > -- > Björn Antonsson > Typesafe <http://typesafe.com/> – Reactive Apps on the JVM > twitter: @bantonsson <http://twitter.com/#!/bantonsson> > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Typesafe <http://typesafe.com/> - Reactive apps on the JVM Twitter: @patriknw -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
