你好，这个报错是任务启动了很长一段时间后才发生的，之前一直正常运行，出现这个报错后就提交不了任务了，感觉是ResourceManager出现了问题，ResourceManager有可能自己死掉吗？ 另外，这个现场还没有关掉，有什么办法排查下jobmaster和ResourceManager是否正常吗？
Xintong Song <tonysong...@gmail.com> 于2020年8月25日周二 上午9:46写道： > 从日志上看是 JobMaster 连不上 ResourceManager。这两个组件应该是在同一个进程内的，通过 akka 进行本地通信。 > 需要看下完整的日志，RM 是否成功启动并注册到了 akka 的 actor system，以及注册的路径是否和 JM 尝试连接的路径一致。 > > > Thank you~ > > Xintong Song > > > > On Mon, Aug 24, 2020 at 3:41 PM song wang <sdlcwangson...@gmail.com> > wrote: > > > 各位老哥， flink > > 运行在yarn上，偶尔报错无法解析ResourceManager地址，可是从对应的host上查找是有flink进程的，请问是什么原因呢？ > > flink 版本1.9.0 > > > > 部分日志如下： > > ``` > > 2020-08-24 15:11:31,566 INFO > org.apache.flink.runtime.jobmaster.JobMaster > > - Could not resolve ResourceManager address > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in 10000 > > ms: > > Ask timed out on [ActorSelection[Anchor(akka://flink/), > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is that > > the recipient actor didn't send a reply.. > > 2020-08-24 15:11:51,606 INFO > org.apache.flink.runtime.jobmaster.JobMaster > > - Could not resolve ResourceManager address > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in 10000 > > ms: > > Ask timed out on [ActorSelection[Anchor(akka://flink/), > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is that > > the recipient actor didn't send a reply.. > > 2020-08-24 15:12:11,645 INFO > org.apache.flink.runtime.jobmaster.JobMaster > > - Could not resolve ResourceManager address > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in 10000 > > ms: > > Ask timed out on [ActorSelection[Anchor(akka://flink/), > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is that > > the recipient actor didn't send a reply.. > > 2020-08-24 15:12:31,687 INFO > org.apache.flink.runtime.jobmaster.JobMaster > > - Could not resolve ResourceManager address > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in 10000 > > ms: > > Ask timed out on [ActorSelection[Anchor(akka://flink/), > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is that > > the recipient actor didn't send a reply.. > > 2020-08-24 15:12:51,727 INFO > org.apache.flink.runtime.jobmaster.JobMaster > > - Could not resolve ResourceManager address > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in 10000 > > ms: > > Ask timed out on [ActorSelection[Anchor(akka://flink/), > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is that > > the recipient actor didn't send a reply.. > > 2020-08-24 15:13:08,198 INFO > > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job > > Streaming WordCount (ff0ab7ec3e577a8e0c69e1c8454e5b72) switched from > state > > RUNNING to FAILING. > > > org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException: > > Could not allocate all requires slots within timeout of 300000 ms. Slots > > required: 9, slots allocated: 0, previous allocation IDs: [], execution > > status: completed exceptionally: > java.util.concurrent.CompletionException: > > java.util.concurrent.CompletionException: > > > > > java.util.concurrent.TimeoutException/java.util.concurrent.CompletableFuture@28d7ad5d > > [Completed > > exceptionally], incomplete: > java.util.concurrent.CompletableFuture@663cdf7e > > [Not > > completed, 1 dependents], incomplete: > > java.util.concurrent.CompletableFuture@2058a7e9[Not completed, 1 > > dependents], incomplete: java.util.concurrent.CompletableFuture@5c1121c8 > > [Not > > completed, 1 dependents], incomplete: > > java.util.concurrent.CompletableFuture@49b9c252[Not completed, 1 > > dependents], incomplete: java.util.concurrent.CompletableFuture@497e3334 > > [Not > > completed, 1 dependents], incomplete: > > java.util.concurrent.CompletableFuture@2c7ca21d[Not completed, 1 > > dependents], incomplete: java.util.concurrent.CompletableFuture@7936c93b > > [Not > > completed, 1 dependents], incomplete: > > java.util.concurrent.CompletableFuture@7e9a2f1d[Not completed, 1 > > dependents] > > at > > > > > org.apache.flink.runtime.executiongraph.SchedulingUtils.lambda$scheduleEager$1(SchedulingUtils.java:194) > > at > > > > > java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) > > at > > > > > java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) > > at > > > > > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > > > > > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > > > > > org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.handleCompletedFuture(FutureUtils.java:633) > > at > > > > > org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.lambda$new$0(FutureUtils.java:656) > > at > > > > > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > > at > > > > > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > > at > > > > > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > > > > > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > > > > > org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:190) > > at > > > > > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > > at > > > > > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > > at > > > > > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > > > > > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > > > > > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:700) > > at > > > > > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:484) > > at > > > > > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:380) > > at > > > > > java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822) > > at > > > > > java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797) > > at > > > > > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > > at > > > > > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > > at > > > > > org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:998) > > at > > > > > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397) > > at > > > > > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190) > > at > > > > > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > > at > > > > > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > > at akka.japi.pf > .UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > > > > > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > > > > > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > ``` > > >
