>>>Ok I missed the log below. I guess when the task manager was stopped this happened. I think if the TM stopped you also would not get the log. But It will throw another "UnknownTaskExecutorException", which would include something like “No TaskExecutor registered under ”.
>>> But I guess it's ok and not a big issue??? Does this happen continuously? Best, Guowei On Thu, Nov 4, 2021 at 12:39 AM John Smith <java.dev....@gmail.com> wrote: > Ok I missed the log below. I guess when the task manager was stopped this > happened. > > I attached the full sequence. But I guess it's ok and not a big issue??? > > > 2021-11-02 23:20:22,682 ERROR > org.apache.flink.runtime.rest.handler.taskmanager. > TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor 7e1 > b7db5918004e4160fdecec1bbdad7. > java.util.concurrent.CompletionException: org.apache.flink.util. > FlinkException: Could not retrieve file from transient blob store. > at org.apache.flink.runtime.rest.handler.taskmanager. > AbstractTaskManagerFileHandler.lambda$respondToRequest$0( > AbstractTaskManagerFileHandler.java:135) > at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture > .java:670) > at java.util.concurrent.CompletableFuture$UniAccept.tryFire( > CompletableFuture.java:646) > at java.util.concurrent.CompletableFuture$Completion.run( > CompletableFuture.java:456) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) > at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop > .run(NioEventLoop.java:515) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) > at org.apache.flink.shaded.netty4.io.netty.util.internal. > ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.flink.util.FlinkException: Could not retrieve file > from transient blob store. > ... 10 more > Caused by: java.io.FileNotFoundException: Local file /tmp/blobStore-9 > cb73f27-11db-4c42-a3fc-9b77f558e722/no_job/blob_t-274d3 > c2d5acd78ced877d898b1877b10b62a64df-590b54325d599a6782a77413691e0a7b does > not exist and failed to copy from blob store. > at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer > .java:516) > at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer > .java:444) > at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java: > 369) > at org.apache.flink.runtime.rest.handler.taskmanager. > AbstractTaskManagerFileHandler.lambda$respondToRequest$0( > AbstractTaskManagerFileHandler.java:133) > ... 9 more > 2021-11-02 23:20:22,703 ERROR > org.apache.flink.runtime.rest.handler.taskmanager. > TaskManagerLogFileHandler - Unhandled exception. > org.apache.flink.util.FlinkException: Could not retrieve file from > transient blob store. > at org.apache.flink.runtime.rest.handler.taskmanager. > AbstractTaskManagerFileHandler.lambda$respondToRequest$0( > AbstractTaskManagerFileHandler.java:135) > at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture > .java:670) > at java.util.concurrent.CompletableFuture$UniAccept.tryFire( > CompletableFuture.java:646) > at java.util.concurrent.CompletableFuture$Completion.run( > CompletableFuture.java:456) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416) > at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop > .run(NioEventLoop.java:515) > at org.apache.flink.shaded.netty4.io.netty.util.concurrent. > SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) > at org.apache.flink.shaded.netty4.io.netty.util.internal. > ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.io.FileNotFoundException: Local file /tmp/blobStore-9 > cb73f27-11db-4c42-a3fc-9b77f558e722/no_job/blob_t-274d3 > c2d5acd78ced877d898b1877b10b62a64df-590b54325d599a6782a77413691e0a7b does > not exist and failed to copy from blob store. > at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer > .java:516) > at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer > .java:444) > at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java: > 369) > at org.apache.flink.runtime.rest.handler.taskmanager. > AbstractTaskManagerFileHandler.lambda$respondToRequest$0( > AbstractTaskManagerFileHandler.java:133) > ... 9 more > > On Wed, 3 Nov 2021 at 02:48, Guowei Ma <guowei....@gmail.com> wrote: > >> Hi, Smith >> >> It seems that the log file(blob_t-274d3c2d5acd78ced877d89 >> 8b1877b10b62a64df-590b54325d599a6782a77413691e0a7b) is deleted for some >> reason. But AFAIK there are no other guys reporting this exception.(Maybe >> other guys know what would happen). >> 1. I think if you could refresh the page and you would see the correct >> result because this would trigger another file retrieving from TM. >> 2. And It might be more safe that setting an dedicated blob >> directory path(other than /tmp) `blob.storage.directory`[1] >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#blob-storage-directory >> >> >> Best, >> Guowei >> >> >> On Wed, Nov 3, 2021 at 7:50 AM John Smith <java.dev....@gmail.com> wrote: >> >>> Hi running Flink 1.10.0 With 3 zookeepers, 3 job nodes and 3 task nodes. >>> and I saw this exception on the job node logs... >>> 2021-11-02 23:20:22,703 ERROR >>> org.apache.flink.runtime.rest.handler.taskmanager. >>> TaskManagerLogFileHandler - Unhandled exception. >>> org.apache.flink.util.FlinkException: Could not retrieve file from >>> transient blob store. >>> at org.apache.flink.runtime.rest.handler.taskmanager. >>> AbstractTaskManagerFileHandler.lambda$respondToRequest$0( >>> AbstractTaskManagerFileHandler.java:135) >>> at java.util.concurrent.CompletableFuture.uniAccept( >>> CompletableFuture.java:670) >>> at java.util.concurrent.CompletableFuture$UniAccept.tryFire( >>> CompletableFuture.java:646) >>> at java.util.concurrent.CompletableFuture$Completion.run( >>> CompletableFuture.java:456) >>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >>> AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) >>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >>> SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416 >>> ) >>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop >>> .run(NioEventLoop.java:515) >>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent. >>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) >>> at org.apache.flink.shaded.netty4.io.netty.util.internal. >>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: java.io.FileNotFoundException: Local file /tmp/blobStore-9 >>> cb73f27-11db-4c42-a3fc-9b77f558e722/no_job/blob_t-274d3 >>> c2d5acd78ced877d898b1877b10b62a64df-590b54325d599a6782a77413691e0a7b >>> does not exist and failed to copy from blob store. >>> at org.apache.flink.runtime.blob.BlobServer.getFileInternal( >>> BlobServer.java:516) >>> at org.apache.flink.runtime.blob.BlobServer.getFileInternal( >>> BlobServer.java:444) >>> at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java: >>> 369) >>> at org.apache.flink.runtime.rest.handler.taskmanager. >>> AbstractTaskManagerFileHandler.lambda$respondToRequest$0( >>> AbstractTaskManagerFileHandler.java:133) >>> ... 9 more >>> >>
