Ok I missed the log below. I guess when the task manager was stopped this
happened.
I attached the full sequence. But I guess it's ok and not a big issue???
2021-11-02 23:20:22,682 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
- Failed to transfer file from TaskExecutor 7e1
b7db5918004e4160fdecec1bbdad7.
java.util.concurrent.CompletionException: org.apache.flink.util.
FlinkException: Could not retrieve file from transient blob store.
at org.apache.flink.runtime.rest.handler.taskmanager.
AbstractTaskManagerFileHandler.lambda$respondToRequest$0(
AbstractTaskManagerFileHandler.java:135)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture
.java:670)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(
CompletableFuture.java:646)
at java.util.concurrent.CompletableFuture$Completion.run(
CompletableFuture.java:456)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(
NioEventLoop.java:515)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.
ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.util.FlinkException: Could not retrieve file
from transient blob store.
... 10 more
Caused by: java.io.FileNotFoundException: Local file /tmp/blobStore-9
cb73f27-11db-4c42-a3fc-9b77f558e722/no_job/blob_t-274d3
c2d5acd78ced877d898b1877b10b62a64df-590b54325d599a6782a77413691e0a7b does
not exist and failed to copy from blob store.
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer
.java:516)
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer
.java:444)
at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java:369)
at org.apache.flink.runtime.rest.handler.taskmanager.
AbstractTaskManagerFileHandler.lambda$respondToRequest$0(
AbstractTaskManagerFileHandler.java:133)
... 9 more
2021-11-02 23:20:22,703 ERROR
org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler
- Unhandled exception.
org.apache.flink.util.FlinkException: Could not retrieve file from transient
blob store.
at org.apache.flink.runtime.rest.handler.taskmanager.
AbstractTaskManagerFileHandler.lambda$respondToRequest$0(
AbstractTaskManagerFileHandler.java:135)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture
.java:670)
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(
CompletableFuture.java:646)
at java.util.concurrent.CompletableFuture$Completion.run(
CompletableFuture.java:456)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(
NioEventLoop.java:515)
at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at org.apache.flink.shaded.netty4.io.netty.util.internal.
ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: Local file /tmp/blobStore-9
cb73f27-11db-4c42-a3fc-9b77f558e722/no_job/blob_t-274d3
c2d5acd78ced877d898b1877b10b62a64df-590b54325d599a6782a77413691e0a7b does
not exist and failed to copy from blob store.
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer
.java:516)
at org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer
.java:444)
at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java:369)
at org.apache.flink.runtime.rest.handler.taskmanager.
AbstractTaskManagerFileHandler.lambda$respondToRequest$0(
AbstractTaskManagerFileHandler.java:133)
... 9 more
On Wed, 3 Nov 2021 at 02:48, Guowei Ma <guowei....@gmail.com> wrote:
> Hi, Smith
>
> It seems that the log file(blob_t-274d3c2d5acd78ced877d89
> 8b1877b10b62a64df-590b54325d599a6782a77413691e0a7b) is deleted for some
> reason. But AFAIK there are no other guys reporting this exception.(Maybe
> other guys know what would happen).
> 1. I think if you could refresh the page and you would see the correct
> result because this would trigger another file retrieving from TM.
> 2. And It might be more safe that setting an dedicated blob
> directory path(other than /tmp) `blob.storage.directory`[1]
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#blob-storage-directory
>
>
> Best,
> Guowei
>
>
> On Wed, Nov 3, 2021 at 7:50 AM John Smith <java.dev....@gmail.com> wrote:
>
>> Hi running Flink 1.10.0 With 3 zookeepers, 3 job nodes and 3 task nodes.
>> and I saw this exception on the job node logs...
>> 2021-11-02 23:20:22,703 ERROR
>> org.apache.flink.runtime.rest.handler.taskmanager.
>> TaskManagerLogFileHandler - Unhandled exception.
>> org.apache.flink.util.FlinkException: Could not retrieve file from
>> transient blob store.
>> at org.apache.flink.runtime.rest.handler.taskmanager.
>> AbstractTaskManagerFileHandler.lambda$respondToRequest$0(
>> AbstractTaskManagerFileHandler.java:135)
>> at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture
>> .java:670)
>> at java.util.concurrent.CompletableFuture$UniAccept.tryFire(
>> CompletableFuture.java:646)
>> at java.util.concurrent.CompletableFuture$Completion.run(
>> CompletableFuture.java:456)
>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>> AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>> SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
>> at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop
>> .run(NioEventLoop.java:515)
>> at org.apache.flink.shaded.netty4.io.netty.util.concurrent.
>> SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
>> at org.apache.flink.shaded.netty4.io.netty.util.internal.
>> ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.io.FileNotFoundException: Local file /tmp/blobStore-9
>> cb73f27-11db-4c42-a3fc-9b77f558e722/no_job/blob_t-274d3
>> c2d5acd78ced877d898b1877b10b62a64df-590b54325d599a6782a77413691e0a7b
>> does not exist and failed to copy from blob store.
>> at org.apache.flink.runtime.blob.BlobServer.getFileInternal(
>> BlobServer.java:516)
>> at org.apache.flink.runtime.blob.BlobServer.getFileInternal(
>> BlobServer.java:444)
>> at org.apache.flink.runtime.blob.BlobServer.getFile(BlobServer.java:
>> 369)
>> at org.apache.flink.runtime.rest.handler.taskmanager.
>> AbstractTaskManagerFileHandler.lambda$respondToRequest$0(
>> AbstractTaskManagerFileHandler.java:133)
>> ... 9 more
>>
>
