Okay, here are the docs for the Akka version we are using: http://doc.akka.io/docs/akka/2.3.14/scala/remoting.html#Lifecycle_and_Failure_Recovery_Model
It says that after a remote deathwatch trigger, the actor system must be restarted before it can connect again. We probably need to do the following: - Either restart TaskManager actor system when it detects that it is quarantined (or when it senses the JobManager as failed) - Or switch from Akka death watch to a manual heartbeat mechanism Would be good to also have Till's input on this... What do you think? Stephan On Thu, Feb 4, 2016 at 5:11 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: > Yes exactly , it says it is quarantined. > > Gyula > > > Gyula > > On Thu, Feb 4, 2016 at 4:09 PM Stephan Ewen <se...@apache.org> wrote: > >> @Gyula Do you see log messages about quarantined actor systems? >> >> There may be an issue with Akka Death watches that once the connection is >> lost, it cannot be re-established unless the TaskManager is restarted >> >> >> http://doc.akka.io/docs/akka/current/scala/remoting.html#Lifecycle_and_Failure_Recovery_Model >> >> >> >> On Thu, Feb 4, 2016 at 5:03 PM, Radu Tudoran <radu.tudo...@huawei.com> >> wrote: >> >>> Hi, >>> >>> >>> >>> Well…yesterday when I looked into it there was no additional info than >>> the one I have send. Today I reproduced the problem and I could see in the >>> log file. >>> >>> >>> >>> >>> >>> akka.actor.ActorInitializationException: exception during creation >>> >>> at >>> akka.actor.ActorInitializationException$.apply(Actor.scala:164) >>> >>> at akka.actor.ActorCell.create(ActorCell.scala:596) >>> >>> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) >>> >>> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) >>> >>> at >>> akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:279) >>> >>> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >>> >>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> >>> Caused by: java.lang.reflect.InvocationTargetException >>> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >>> >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> >>> at >>> java.lang.reflect.Constructor.newInstance(Constructor.java:422) >>> >>> at akka.util.Reflect$.instantiate(Reflect.scala:66) >>> >>> at akka.actor.ArgsReflectConstructor.produce(Props.scala:352) >>> >>> at akka.actor.Props.newActor(Props.scala:252) >>> >>> at akka.actor.ActorCell.newActor(ActorCell.scala:552) >>> >>> at akka.actor.ActorCell.create(ActorCell.scala:578) >>> >>> ... 10 more >>> >>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> 11:21:17,423 ERROR >>> org.apache.flink.runtime.taskmanager.Task >>> - FATAL - exception in task resource cleanup >>> >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> 11:21:55,160 ERROR >>> org.apache.flink.runtime.taskmanager.Task >>> - FATAL - exception in task exception handler >>> >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> >>> >>> …. >>> >>> >>> >>> - Unexpected exception in the selector loop. >>> >>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>> >>> >>> >>> >>> >>> Looks like the input flow is faster than the GC collector >>> >>> >>> >>> Dr. Radu Tudoran >>> >>> Research Engineer - Big Data Expert >>> >>> IT R&D Division >>> >>> >>> >>> [image: cid:image007.jpg@01CD52EB.AD060EE0] >>> >>> HUAWEI TECHNOLOGIES Duesseldorf GmbH >>> >>> European Research Center >>> >>> Riesstrasse 25, 80992 München >>> >>> >>> >>> E-mail: *radu.tudo...@huawei.com <radu.tudo...@huawei.com>* >>> >>> Mobile: +49 15209084330 >>> >>> Telephone: +49 891588344173 >>> >>> >>> >>> HUAWEI TECHNOLOGIES Duesseldorf GmbH >>> Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com >>> Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063, >>> Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN >>> Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063, >>> Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN >>> >>> This e-mail and its attachments contain confidential information from >>> HUAWEI, which is intended only for the person or entity whose address is >>> listed above. Any use of the information contained herein in any way >>> (including, but not limited to, total or partial disclosure, reproduction, >>> or dissemination) by persons other than the intended recipient(s) is >>> prohibited. If you receive this e-mail in error, please notify the sender >>> by phone or email immediately and delete it! >>> >>> >>> >>> *From:* Till Rohrmann [mailto:trohrm...@apache.org] >>> *Sent:* Thursday, February 04, 2016 4:55 PM >>> *To:* user@flink.apache.org >>> *Subject:* Re: release of task slot >>> >>> >>> >>> Hi Radu, >>> >>> what does the log of the TaskManager 10.204.62.80:57910 say? >>> >>> Cheers, >>> Till >>> >>> >>> >>> >>> >>> On Wed, Feb 3, 2016 at 6:00 PM, Radu Tudoran <radu.tudo...@huawei.com> >>> wrote: >>> >>> Hello, >>> >>> >>> >>> >>> >>> I am facing an error which for which I cannot figure the cause. Any idea >>> what could cause such an error? >>> >>> >>> >>> >>> >>> >>> >>> java.lang.Exception: The slot in which the task was executed has been >>> released. Probably loss of TaskManager a8b69bd9449ee6792e869a9ff9e843e2 @ >>> cloudr6-admin - 4 slots - URL: akka.tcp:// >>> flink@10.204.62.80:57910/user/taskmanager >>> >>> at >>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:151) >>> >>> at >>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547) >>> >>> at >>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119) >>> >>> at >>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156) >>> >>> at >>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215) >>> >>> at >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696) >>> >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) >>> >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) >>> >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) >>> >>> at >>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44) >>> >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) >>> >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) >>> >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) >>> >>> at >>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33) >>> >>> at >>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28) >>> >>> at >>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) >>> >>> at >>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28) >>> >>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>> >>> at >>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100) >>> >>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>> >>> at >>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46) >>> >>> at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369) >>> >>> at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501) >>> >>> at akka.actor.ActorCell.invoke(ActorCell.scala:486) >>> >>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) >>> >>> at akka.dispatch.Mailbox.run(Mailbox.scala:221) >>> >>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> >>> at >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> >>> >>> >>> >>> >>> Dr. Radu Tudoran >>> >>> Research Engineer - Big Data Expert >>> >>> IT R&D Division >>> >>> >>> >>> [image: cid:image007.jpg@01CD52EB.AD060EE0] >>> >>> HUAWEI TECHNOLOGIES Duesseldorf GmbH >>> >>> European Research Center >>> >>> Riesstrasse 25, 80992 München >>> >>> >>> >>> E-mail: *radu.tudo...@huawei.com <radu.tudo...@huawei.com>* >>> >>> Mobile: +49 15209084330 >>> >>> Telephone: +49 891588344173 >>> >>> >>> >>> HUAWEI TECHNOLOGIES Duesseldorf GmbH >>> Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com >>> Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063, >>> Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN >>> Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063, >>> Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN >>> >>> This e-mail and its attachments contain confidential information from >>> HUAWEI, which is intended only for the person or entity whose address is >>> listed above. Any use of the information contained herein in any way >>> (including, but not limited to, total or partial disclosure, reproduction, >>> or dissemination) by persons other than the intended recipient(s) is >>> prohibited. If you receive this e-mail in error, please notify the sender >>> by phone or email immediately and delete it! >>> >>> >>> >>> >>> >> >>
