Yes I forgot to mention in my first email. I have tried increasing taskmanager.network.request-backoff.max to 30000 in flink-conf.yaml. But I am getting the same error.
On Wed, Jul 14, 2021 at 7:10 PM Timo Walther <twal...@apache.org> wrote: > Hi Debraj, > > I could find quite a few older emails that were suggesting to play > around with the `taskmanager.network.request-backoff.max` option. This > was also recomended in the link that you shared. Have you tried it? > > Here is some background: > > > http://deprecated-apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Network-PartitionNotFoundException-when-run-on-multi-nodes-td21683.html > > https://issues.apache.org/jira/browse/FLINK-9413 > > Let us know if it helped. > > Regards, > Timo > > > > On 14.07.21 14:51, Debraj Manna wrote: > > Hi > > > > I am observing my flink jobs is failing with the below error > > > > 2021-07-14T12:07:00.918Z INFO runtime.executiongraph.Execution > > flink-akka.actor.default-dispatcher-29 transitionState:1446 > > MetricAggregateFunction -> (Sink: LateMetricSink10, Sink: TSDBSink9) > > (12/30) (3489393394c13fd1ad85136e11d67deb) switched from RUNNING to > > FAILED on container_e12_1626252668574_0001_01_000005 @ platform4 > > (dataPort=34313). > > > > org.apache.flink.runtime.io.network.partition.PartitionNotFoundException: > Partition > > 73b5c8ee0c6db91027ce081379f64669#6@7c77e83642e1b954a91dd7b49cf3043b not > > found. > > > > at > > org.apache.flink.runtime.io > .network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:280) > > > > at > > org.apache.flink.runtime.io > .network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:198) > > > > at > > org.apache.flink.runtime.io > .network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:515) > > > > at > > org.apache.flink.runtime.io > .network.partition.consumer.SingleInputGate.lambda$triggerPartitionStateCheck$1(SingleInputGate.java:845) > > > > at > > > java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) > > > > at > > > java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) > > > > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) > > > > at > > > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) > > > > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > > > at > > > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > > > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > > > at > > > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > > > > > Same has been discussed in the old link. But in my case there is hardly > > any load in the cluster. Can someone suggest what can cause this error? > > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html > > < > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/PartitionNotFoundException-when-running-in-yarn-session-td16081.html > > > > > > > > Thanks, > > > > > >
