Hello, good catch - we are running in this situation in the moment as well - some of our sharded actors register to a Cluster Pub-Sub topic when started - when getting rebalanced or when a node crashes the sharded actor is not restarted and hence not subscribed anymore... First thought would also be to just extend ClusterSharding.start with a keepAlive field to restart actors automatically? For now, I think we will solve this by implementing a Watchdog as ClusterSingleton.
michael On Friday, 16 May 2014 09:25:05 UTC+2, Patrik Nordwall wrote: > > Hi Jeroen, > > > On Thu, May 15, 2014 at 10:09 PM, Jeroen Gordijn > <jeroen....@gmail.com<javascript:> > > wrote: > >> Hi, >> >> When rebalancing a Shard, the old shard is stopped and a new shard is >> started on another node, after which all messages for that Shard will be >> send to the new node. When a message is received, the actor will be >> created. When Akka-persistence is used the Actor will reload all its events >> and restore state before processing the new message. But if no message is >> sent, the actor will not be created. This can be problematic when the actor >> is has some active state with retry mechanisme or timeout. Is my >> understanding correct? >> > > Your reasoning is correct. I think you can implement that by letting the > actor schedule a keep-alive message to itself, but via the ShardRegion. > Normally that will be local only message roundtrip via the scheduler and > local ShardRegion, but after rebalancing it will delegate the message to > the new node and thereby wake up the actor again. > > What this doesn't solve is when a node crashes. An actor living on that > node will not be automatically started somewhere else, until a message is > sent to it. To solve that I think you have to let the actor register itself > to a a few (for redundancy) watchdog actors, which watch the actor and know > how to send the wake-up message via ClusterSharding. > > Does that make sense? > > Cheers, > Patrik > > >> >> Is there a way to actively restore the Shard state when the shard is >> moved to another node? One problem I can see with this is when going back >> to less nodes. This means that the shards will be rebalanced, but >> potentially giving memory problems. This will cause rebalancing and memory >> problems on the next node and eventually putting the whole cluster down. >> Starting the cluster will be also problematic for the same reason. >> >> Cheers, >> Jeroen >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+...@googlegroups.com <javascript:>. >> To post to this group, send email to akka...@googlegroups.com<javascript:> >> . >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > JOIN US. REGISTER TODAY! <http://www.scaladays.org/> > Scala <http://www.scaladays.org/> > Days <http://www.scaladays.org/> > June 16th-18th, <http://www.scaladays.org/> > Berlin <http://www.scaladays.org/> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
