> User can install supervision that catch and for example resume, which can result in that the in-memory state is different from what is actually stored.
Sure, the user can catch all Throwables, but my statement was that it doesn't happen by default. > We have a solution that works. Why change? I'm not necessarily suggesting that we have to change it right now. This is more of a discussion/exploration of an alternative to overloading the meaning of stopping an actor as a failure for akka persistence. Roland mentioned that you guys are "making all lifecycle events (like child failure) normal messages in Akka Typed" (https://github.com/akka/akka/pull/18776#discussion_r43050078). I think also opens the path to an alternative to having the persistent actor stop itself (my assumption here is that supervisors in akka typed are going to be explicit actors rather than the parents being implicit supervisors). On Thursday, December 17, 2015 at 10:15:10 PM UTC-8, Patrik Nordwall wrote: > > Ah, I missed that you extended Throwable. > That will anyway not enforce that the actor is always stopped when there > are storage exceptions. User can install supervision that catch and for > example resume, which can result in that the in-memory state is different > from what is actually stored. > > We have a solution that works. Why change? I'm sorry that I'm not open for > more flexibility in this error handling, but it's no fun to take > responsibility for support questions about data corruption. > tors 17 dec. 2015 kl. 22:10 skrev Henry Mai <henry....@gmail.com > <javascript:>>: > >> That is not true. >> The default behavior to restart does not apply to custom exceptions that >> extend from Throwable. It will escalate custom exceptions that extend from >> Throwable. >> You can test this out yourself by creating a parent actor and a child >> actor within the parent actor and have it throw something extending from a >> Throwable. >> >> Here is an example that demonstrates that it escalates and causes the >> system to shutdown: >> >> Code: >> package somepackage >> >> import akka.actor._ >> >> class CustomException extends Throwable >> >> class B extends Actor { >> self ! "Throw" >> >> def receive: Receive = { >> case "Throw" => throw new CustomException >> } >> } >> >> >> class A extends Actor { >> val child = context.actorOf(Props(new B)) >> >> def receive: Receive = { >> case _ => >> } >> } >> >> object Main extends App { >> val system = ActorSystem() >> system.actorOf(Props(new A)) >> } >> >> >> Output: >> [info] Running somepackage.Main >> [ERROR] [12/17/2015 13:09:51.738] >> [default-akka.actor.default-dispatcher-3] >> [LocalActorRefProvider(akka://default)] guardian failed, shutting down >> system >> somepackage.CustomException >> at somepackage.B$$anonfun$receive$1.applyOrElse(A.scala:11) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:467) >> at somepackage.B.aroundReceive(A.scala:7) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) >> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> [ERROR] [12/17/2015 13:09:51.739] >> [default-akka.actor.default-dispatcher-3] [akka://default/user] null >> somepackage.CustomException >> at somepackage.B$$anonfun$receive$1.applyOrElse(A.scala:11) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:467) >> at somepackage.B.aroundReceive(A.scala:7) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) >> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> [success] Total time: 4 s, completed Dec 17, 2015 1:09:51 PM >> >> >> >> On Thursday, December 17, 2015 at 12:55:36 PM UTC-8, Patrik Nordwall >> wrote: >> >>> >>> >>> On Thu, Dec 17, 2015 at 6:14 PM, Henry Mai <henry....@gmail.com> wrote: >>> >>>> There is an alternative to not doing the context.stop(self) on persist >>>> failures, that I didn't follow up on in the discussion in the pull >>>> request; >>>> you can throw a special exception that inherits from Throwable (let's call >>>> this `PersistFailureException`) >>>> >>> >>> That is almost what we had in 2.3, and that does not ensure that the >>> actor is stopped unconditionally. By default it will be restarted. >>> >>> >>>> >>>> So, the motivation behind context.stop(self) on a persist failure is >>>> that you don't want to have to worry about the child actor stopping itself >>>> correctly (to guarantee safety). Additionally we also know that the actor >>>> can be restarted (without having to be unconditionally stopped), as long >>>> as >>>> it's done with an exponential backoff. >>>> >>>> Therefore by throwing `PersistFailureException` we can guarantee that >>>> by default, if the developer does not have a supervisor strategy that >>>> handles this special exception, it will bubble up all the way to the root >>>> guardian and stop the entire actor system. >>>> >>> >>> That is not how default supervision works. By default the actor will be >>> restarted. >>> >>> >>>> So now instead of the documentation saying that the Persistent Actor >>>> always shuts itself down, it will say: >>>> "Your SupervisorStrategy must handle the `PersistFailureException`, >>>> either by specifying a `Stop` directive on a normal supervisor, or by >>>> using >>>> the `Restart` directive under the >>>> `TransparentExponentialBackoffSupervisor`. Otherwise if left unhandled >>>> this >>>> exception will bubble all the way up to the root guardian and cause the >>>> ActorSystem to shutdown." >>>> >>>> >>>> On Thursday, December 17, 2015 at 3:18:47 AM UTC-8, Patrik Nordwall >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Dec 17, 2015 at 11:58 AM, Raymond Roestenburg < >>>>> raymond.r...@gmail.com> wrote: >>>>> >>>>>> Hi Patrik, >>>>>> >>>>>> Henry Mai looped me in on the existence of >>>>>> the TransparentExponentialBackoffSupervisor in akka-contrib and PR >>>>>> https://github.com/akka/akka/pull/18776 (thanks Henry!). >>>>>> I find it confusing that there are two. As a user I would like one >>>>>> BackoffSupervisor, in akka.pattern, not two, one of which is in contrib >>>>>> (I >>>>>> know, users always want the impossible!). >>>>>> >>>>> >>>>> Me too. I tried, but failed. It was motivated by that they are >>>>> different, and should be different classes. >>>>> >>>>> >>>>>> >>>>>> This is a very common pattern that pops up all the time which >>>>>> deserves a spot in akka.pattern IMHO. >>>>>> Almost any time you're working with some network you can't just use >>>>>> the default supervisor stuff since it causes too fast reconnects, I >>>>>> think >>>>>> it should be part of akka-actor, not akka-contrib. >>>>>> >>>>>> Both have the issue I was referring to, so I could implement the fix >>>>>> in both now, sidestepping the issue of wanting just one version. >>>>>> >>>>>> But let me apply some logic, (subjective, possibly flawed :-) >>>>>> >>>>>> Are there any plans (or even possibility) to change the 'Stop >>>>>> behavior' in akka persistence? >>>>>> >>>>> >>>>> Why? We had flawed failure handling in Akka Persistence 2.3. We must >>>>> enforce stopping of the persistent actor when there are journal failures, >>>>> becuase the state must not diverge from what has actually been stored. >>>>> http://doc.akka.io/docs/akka/2.4.1/scala/persistence.html#Failures >>>>> >>>>> >>>>>> >>>>>> If it does not, I'm assuming everyone agrees that Stop is a good >>>>>> option to indicate failure (in certain scenarios), which I think means >>>>>> that >>>>>> there is a good reason to have options on one implementation. >>>>>> >>>>>> If Stop is not good to indicate failure, I would expect >>>>>> akka-persistence to change, which also means there is just one version >>>>>> of >>>>>> the BackoffSupervisor. >>>>>> >>>>>> But then, what happens if someone puts this in front of a remote >>>>>> actor, where deathwatch (i.e Stop) is the only option to indicate >>>>>> failure?... >>>>>> >>>>>> "Location transparency, anyone?" ;-) >>>>>> >>>>>> Cheers, >>>>>> Ray >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Cheers, >>>>>> Ray >>>>>> >>>>>> On Wednesday, December 16, 2015 at 9:30:43 PM UTC+2, Raymond >>>>>> Roestenburg wrote: >>>>>>> >>>>>>> Ok, I'll put a PR in for this. >>>>>>> >>>>>>> Cheers, >>>>>>> Ray >>>>>>> >>>>>>> On Wednesday, December 16, 2015 at 7:09:59 PM UTC+2, Patrik Nordwall >>>>>>> wrote: >>>>>>>> >>>>>>>> Sounds like a good improvement. PR is very welcome. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Patrik >>>>>>>> >>>>>>>> On Tue, Dec 15, 2015 at 1:31 PM, Raymond Roestenburg < >>>>>>>> raymond.r...@gmail.com> wrote: >>>>>>>> >>>>>>>>> BTW if you think this is a good idea, I would be happy to provide >>>>>>>>> a PR for it (since I can't move to 2.4.1 yet anyway and I'm >>>>>>>>> back-porting it >>>>>>>>> right now). >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tuesday, December 15, 2015 at 2:28:36 PM UTC+2, Raymond >>>>>>>>> Roestenburg wrote: >>>>>>>>>> >>>>>>>>>> Hey hAkkers, >>>>>>>>>> >>>>>>>>>> I'm trying out the new BackoffSupervisor (in 2.4.1) and there is >>>>>>>>>> one thing that I did not really like (and please correct me if I am >>>>>>>>>> wrong): >>>>>>>>>> >>>>>>>>>> There seems to be no way for the child to tell the >>>>>>>>>> BackoffSupervisor that everything is ok again, the back-off is >>>>>>>>>> automatically reset based on a duration of no errors: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://github.com/akka/akka/blob/v2.4.1/akka-actor/src/main/scala/akka/pattern/BackoffSupervisor.scala#L204 >>>>>>>>>> >>>>>>>>>> https://github.com/akka/akka/blob/v2.4.1/akka-actor/src/main/scala/akka/pattern/BackoffSupervisor.scala#L207-L208 >>>>>>>>>> >>>>>>>>>> The minBackoff is both used as minimal back-off duration and as >>>>>>>>>> this automatic reset duration. Which actually makes the minBackoff >>>>>>>>>> hard to >>>>>>>>>> set correctly, since it depends on the types of errors (and the >>>>>>>>>> speed at >>>>>>>>>> which the errors occur) in the child, and it makes it a bad idea to >>>>>>>>>> set it >>>>>>>>>> too low, since the back-off can then get stuck in repeating per >>>>>>>>>> minBackoff >>>>>>>>>> if the child does not crash fast enough the first time (within >>>>>>>>>> minBackoff), >>>>>>>>>> which means it will not properly back-off. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> So in my case the child is something that connects over the >>>>>>>>>> network and might crash, which could be a very fast error, or a very >>>>>>>>>> slow >>>>>>>>>> error. >>>>>>>>>> A separate duration for reset would be a simple improvement, or >>>>>>>>>> even better a message the child can send to the BackoffSupervisor to >>>>>>>>>> indicate success (basically BackoffSupervisor .ResetRestartCount) >>>>>>>>>> and a way >>>>>>>>>> to disable the auto-reset at minBackoff. >>>>>>>>>> >>>>>>>>>> (I do appreciate that the BackoffSupervisor is done in such a way >>>>>>>>>> that it can be dropped in between actors with hardly any change!) >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Ray >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>>> https://groups.google.com/group/akka-user >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Akka User List" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Patrik Nordwall >>>>>>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>>>>>>> Twitter: @patriknw >>>>>>>> >>>>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: >>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: >>>>>> https://groups.google.com/group/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to akka-user+...@googlegroups.com. >>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Patrik Nordwall >>>>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>>>> Twitter: @patriknw >>>>> >>>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+...@googlegroups.com. >>>> To post to this group, send email to akka...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+...@googlegroups.com <javascript:>. >> To post to this group, send email to akka...@googlegroups.com >> <javascript:>. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
