[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-29234:
---
Labels: pull-request-available (was: )
> Dead lock in DefaultLeaderElectionService
> -
>
> Key: FLINK-29234
> URL: https://issues.apache.org/jira/browse/FLINK-29234
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.5, 1.14.5, 1.15.2
>Reporter: Yu Wang
>Assignee: Weijie Guo
>Priority: Critical
> Labels: pull-request-available
>
> Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
> The log stopped at
> {code:java}
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
> Stopping DefaultLeaderElectionService. {code}
> Which may similar to this ticket
> https://issues.apache.org/jira/browse/FLINK-20008
> Here is the jstack info
> {code:java}
> Found one Java-level deadlock: =
> "flink-akka.actor.default-dispatcher-18": waiting to lock monitor
> 0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
> held by "main-EventThread" "main-EventThread": waiting to lock monitor
> 0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
> held by "flink-akka.actor.default-dispatcher-18" Java stack information for
> the threads listed above: ===
> "flink-akka.actor.default-dispatcher-18": at
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
> - waiting to lock <0x000678d395e8> (a java.lang.Object) at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
> Source) at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
> Source) at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> at
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
> at
> java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
> - locked <0x000678cf1be0> (a java.lang.Object) at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268) at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
> Source) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123) at
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) at
> akka.actor.Actor.aroundReceive(Actor.scala:517) at
> akka.actor.Actor.aroundReceive$(Actor.scala:515) at
> akka.actor.AbstractActor.aroundReceive(A
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-29234:
--
Description:
Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
The log stopped at
{code:java}
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Stopping DefaultLeaderElectionService. {code}
Which may similar to this ticket
https://issues.apache.org/jira/browse/FLINK-20008
Here is the jstack info
{code:java}
Found one Java-level deadlock: =
"flink-akka.actor.default-dispatcher-18": waiting to lock monitor
0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
held by "main-EventThread" "main-EventThread": waiting to lock monitor
0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
held by "flink-akka.actor.default-dispatcher-18" Java stack information for the
threads listed above: ===
"flink-akka.actor.default-dispatcher-18": at
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
- waiting to lock <0x000678d395e8> (a java.lang.Object)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
Source)
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
at
org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
at
java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
- locked <0x000678cf1be0> (a java.lang.Object)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
at org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268)
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
Source)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"main-EventThread":
at
org.apache.flink.runtime.j
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-29234:
--
Description:
Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
The log stopped at
{code:java}
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Stopping DefaultLeaderElectionService. {code}
Which may similar to this ticket
https://issues.apache.org/jira/browse/FLINK-20008
Here is the jstack info
{code:java}
Found one Java-level deadlock:
=
"flink-akka.actor.default-dispatcher-18": waiting to lock monitor
0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
held by "main-EventThread" "main-EventThread": waiting to lock monitor
0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
held by "flink-akka.actor.default-dispatcher-18" Java stack information for the
threads listed above:
===
"flink-akka.actor.default-dispatcher-18":
at
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
- waiting to lock <0x000678d395e8> (a java.lang.Object)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
Source)
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
at
org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
Source)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
at
java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
- locked <0x000678cf1be0> (a java.lang.Object)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
at org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268)
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
Source)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
at akka.actor.Actor.aroundReceive(Actor.scala:517)
at akka.actor.Actor.aroundReceive$(Actor.scala:515)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"main-EventThread":
at
org.apache.flink.runtim
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabian Paul updated FLINK-29234:
Affects Version/s: 1.15.3
(was: 1.15.2)
> Dead lock in DefaultLeaderElectionService
> -
>
> Key: FLINK-29234
> URL: https://issues.apache.org/jira/browse/FLINK-29234
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.5, 1.14.5, 1.15.3
>Reporter: Yu Wang
>Assignee: Weijie Guo
>Priority: Critical
> Labels: pull-request-available
>
> Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
> The log stopped at
> {code:java}
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
> Stopping DefaultLeaderElectionService. {code}
> Which may similar to this ticket
> https://issues.apache.org/jira/browse/FLINK-20008
> Here is the jstack info
> {code:java}
> Found one Java-level deadlock:
> =
> "flink-akka.actor.default-dispatcher-18": waiting to lock monitor
> 0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
> held by "main-EventThread" "main-EventThread": waiting to lock monitor
> 0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
> held by "flink-akka.actor.default-dispatcher-18" Java stack information for
> the threads listed above:
> ===
> "flink-akka.actor.default-dispatcher-18":
> at
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
> - waiting to lock <0x000678d395e8> (a java.lang.Object)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
> Source)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
> Source)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> at
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
> at
> java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
> - locked <0x000678cf1be0> (a java.lang.Object)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
> at org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268)
> at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
> Source)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:517)
> at akka.actor.Actor.aroundReceive$(Actor.scala:515)
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated FLINK-29234:
Priority: Critical (was: Major)
> Dead lock in DefaultLeaderElectionService
> -
>
> Key: FLINK-29234
> URL: https://issues.apache.org/jira/browse/FLINK-29234
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.5
>Reporter: Yu Wang
>Priority: Critical
>
> Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
> Which may similar to this ticket
> https://issues.apache.org/jira/browse/FLINK-20008
> Here is the jstak info
> {code:java}
> Found one Java-level deadlock: =
> "flink-akka.actor.default-dispatcher-18": waiting to lock monitor
> 0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
> held by "main-EventThread" "main-EventThread": waiting to lock monitor
> 0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
> held by "flink-akka.actor.default-dispatcher-18" Java stack information for
> the threads listed above: ===
> "flink-akka.actor.default-dispatcher-18": at
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
> - waiting to lock <0x000678d395e8> (a java.lang.Object) at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
> Source) at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
> Source) at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> at
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
> at
> java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
> - locked <0x000678cf1be0> (a java.lang.Object) at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268) at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
> Source) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123) at
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) at
> akka.actor.Actor.aroundReceive(Actor.scala:517) at
> akka.actor.Actor.aroundReceive$(Actor.scala:515) at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at
> akka.actor.ActorCell.invoke(ActorCell.scala:561) at
> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at
> akka.dispatch.Mailbox.run(Mailbox.scala:225) at
> akka.dispatch.Ma
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated FLINK-29234:
Description:
Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
Which may similar to this ticket
https://issues.apache.org/jira/browse/FLINK-20008
Here is the jstak info
{code:java}
Found one Java-level deadlock: =
"flink-akka.actor.default-dispatcher-18": waiting to lock monitor
0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
held by "main-EventThread" "main-EventThread": waiting to lock monitor
0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
held by "flink-akka.actor.default-dispatcher-18" Java stack information for the
threads listed above: ===
"flink-akka.actor.default-dispatcher-18": at
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
- waiting to lock <0x000678d395e8> (a java.lang.Object) at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
Source) at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
at
org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
Source) at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
at
java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
- locked <0x000678cf1be0> (a java.lang.Object) at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
at org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268)
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
Source) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
scala.PartialFunction.applyOrElse(PartialFunction.scala:123) at
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) at
akka.actor.Actor.aroundReceive(Actor.scala:517) at
akka.actor.Actor.aroundReceive$(Actor.scala:515) at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at
akka.actor.ActorCell.invoke(ActorCell.scala:561) at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at
akka.dispatch.Mailbox.run(Mailbox.scala:225) at
akka.dispatch.Mailbox.exec(Mailbox.scala:235) at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"main-EventThread": at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.runIfStateRunning(JobMasterServiceLeadershipRunner.java:468)
- waiting to lock <0x000678cf1be0> (a java.lang.Object) at
org.
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated FLINK-29234:
Description:
Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
The log stopped at
{code:java}
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
Stopping DefaultLeaderElectionService. {code}
Which may similar to this ticket
https://issues.apache.org/jira/browse/FLINK-20008
Here is the jstack info
{code:java}
Found one Java-level deadlock: =
"flink-akka.actor.default-dispatcher-18": waiting to lock monitor
0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
held by "main-EventThread" "main-EventThread": waiting to lock monitor
0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
held by "flink-akka.actor.default-dispatcher-18" Java stack information for the
threads listed above: ===
"flink-akka.actor.default-dispatcher-18": at
org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
- waiting to lock <0x000678d395e8> (a java.lang.Object) at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
Source) at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
at
org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
Source) at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
at
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
at
java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
at
org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
at
org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
- locked <0x000678cf1be0> (a java.lang.Object) at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
at
org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
at org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268)
at
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
Source) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
scala.PartialFunction.applyOrElse(PartialFunction.scala:123) at
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) at
akka.actor.Actor.aroundReceive(Actor.scala:517) at
akka.actor.Actor.aroundReceive$(Actor.scala:515) at
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at
akka.actor.ActorCell.invoke(ActorCell.scala:561) at
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at
akka.dispatch.Mailbox.run(Mailbox.scala:225) at
akka.dispatch.Mailbox.exec(Mailbox.scala:235) at
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"main-EventThread": at
org.apache.flink.runtime.jobmaster.JobMasterS
[jira] [Updated] (FLINK-29234) Dead lock in DefaultLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-29234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Wang updated FLINK-29234:
Affects Version/s: 1.15.2
1.14.5
> Dead lock in DefaultLeaderElectionService
> -
>
> Key: FLINK-29234
> URL: https://issues.apache.org/jira/browse/FLINK-29234
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.5, 1.14.5, 1.15.2
>Reporter: Yu Wang
>Priority: Critical
>
> Jobmanager stop working because the deadlock in DefaultLeaderElectionService.
> The log stopped at
> {code:java}
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService [] -
> Stopping DefaultLeaderElectionService. {code}
> Which may similar to this ticket
> https://issues.apache.org/jira/browse/FLINK-20008
> Here is the jstack info
> {code:java}
> Found one Java-level deadlock: =
> "flink-akka.actor.default-dispatcher-18": waiting to lock monitor
> 0x7f15c7eae3a8 (object 0x000678d395e8, a java.lang.Object), which is
> held by "main-EventThread" "main-EventThread": waiting to lock monitor
> 0x7f15a3811258 (object 0x000678cf1be0, a java.lang.Object), which is
> held by "flink-akka.actor.default-dispatcher-18" Java stack information for
> the threads listed above: ===
> "flink-akka.actor.default-dispatcher-18": at
> org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService.stop(DefaultLeaderElectionService.java:104)
> - waiting to lock <0x000678d395e8> (a java.lang.Object) at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.lambda$closeAsync$0(JobMasterServiceLeadershipRunner.java:147)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner$$Lambda$735/1742012752.run(Unknown
> Source) at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$$Lambda$736/6716561.accept(Unknown
> Source) at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
> at
> org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
> at
> java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:795)
> at
> java.util.concurrent.CompletableFuture.whenCompleteAsync(CompletableFuture.java:2163)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwardsAsync(FutureUtils.java:684)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.runAfterwards(FutureUtils.java:651)
> at
> org.apache.flink.runtime.jobmaster.JobMasterServiceLeadershipRunner.closeAsync(JobMasterServiceLeadershipRunner.java:143)
> - locked <0x000678cf1be0> (a java.lang.Object) at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateJob(Dispatcher.java:807)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobs(Dispatcher.java:799)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.terminateRunningJobsAndGetTerminationFuture(Dispatcher.java:812)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.onStop(Dispatcher.java:268) at
> org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor$$Lambda$444/1289054037.apply(Unknown
> Source) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at
> akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at
> scala.PartialFunction.applyOrElse(PartialFunction.scala:123) at
> scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) at
> akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at
> scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) at
> akka.actor.Actor.aroundReceive(Actor.scala:517) at
> akka.actor.Actor.aroundReceive$(Actor.scala:515) at
> akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at
> akka.actor.ActorCell.receiveMessage(ActorCell.sc
