[
https://issues.apache.org/jira/browse/FLINK-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580785#comment-14580785
]
Maximilian Michels commented on FLINK-2183:
-------------------------------------------
Here is the stacktrace isolated for the two deadlocked processes. The job
manager holds a lock to release slots when the Disconnect message from the task
manager arrives. It then cancels Executions (in separate processes) running in
the slots, which mark their slots as cancelled. Then they try to release the
slots by acquiring the same lock. However, they cannot do so because the job
manager process already has acquired this lock and tries to release the its
slots. The job manager can never do so because the slots have already been
marked as cancelled by the asynchronous Execution processes.
It seems not so hard to fix but we have to get all the implications right.
Unfortunately I'm running out of time today.
{code}
"flink-akka.actor.default-dispatcher-4" #22 daemon prio=5 os_prio=31
tid=0x00007ff9ccad4000 nid=0x6d03 runnable [0x000000012e844000]
java.lang.Thread.State: RUNNABLE
at
org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:537)
- locked <0x000000076d6bc9d0> (a java.lang.Object)
at
org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
at
org.apache.flink.runtime.instance.Instance.markDead(Instance.java:150)
at
org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:182)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:435)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at
org.apache.flink.runtime.testingUtils.TestingJobManager$$anonfun$receiveTestingMessages$1.applyOrElse(TestingJobManager.scala:185)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:36)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:29)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at
org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:29)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:94)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
at akka.dispatch.Mailbox.run(Mailbox.scala:221)
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"ForkJoinPool-1-worker-13" #40 daemon prio=5 os_prio=31 tid=0x00007ff9ca92b800
nid=0x8903 waiting for monitor entry [0x000000012f669000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSimpleSlot(SlotSharingGroupAssignment.java:478)
- waiting to lock <0x000000076d6bc9d0> (a java.lang.Object)
at
org.apache.flink.runtime.instance.SharedSlot.releaseChild(SharedSlot.java:196)
at
org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:166)
at
org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:734)
at
org.apache.flink.runtime.executiongraph.Execution.fail(Execution.java:565)
at
org.apache.flink.runtime.executiongraph.Execution$4.onComplete(Execution.java:821)
at akka.dispatch.OnComplete.internal(Future.scala:246)
at akka.dispatch.OnComplete.internal(Future.scala:244)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"ForkJoinPool-1-worker-1" #39 daemon prio=5 os_prio=31 tid=0x00007ff9ca92b000
nid=0x8703 waiting for monitor entry [0x000000012f566000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSimpleSlot(SlotSharingGroupAssignment.java:478)
- waiting to lock <0x000000076d6bc9d0> (a java.lang.Object)
at
org.apache.flink.runtime.instance.SharedSlot.releaseChild(SharedSlot.java:196)
at
org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:166)
at
org.apache.flink.runtime.executiongraph.Execution.processFail(Execution.java:734)
at
org.apache.flink.runtime.executiongraph.Execution.fail(Execution.java:565)
at
org.apache.flink.runtime.executiongraph.Execution$4.onComplete(Execution.java:821)
at akka.dispatch.OnComplete.internal(Future.scala:246)
at akka.dispatch.OnComplete.internal(Future.scala:244)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
{code}
> TaskManagerFailsWithSlotSharingITCase fails.
> --------------------------------------------
>
> Key: FLINK-2183
> URL: https://issues.apache.org/jira/browse/FLINK-2183
> Project: Flink
> Issue Type: Bug
> Reporter: Sachin Goel
>
> Travis build is failing on this test.
> Here is the log output:
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/65871133/log.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
