[ https://issues.apache.org/jira/browse/FLINK-17444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105372#comment-17105372 ]
Kostas Kloudas edited comment on FLINK-17444 at 5/12/20, 12:19 PM: ------------------------------------------------------------------- What [~trohrmann] says is correct and it seems that this is the case since the introduction of the AzureFS support in Flink. So the AzureFS was never supported by the {{StreamingFileSink}}. For the {{StreamingFileSink}} to work with the {{HadoopRecoverableWriter}}, the writer should be in the classpath and the check [here|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L60] should be changed to also accept {{wasb}}. That said, I am not sure about the guarantees (e.g. consistency) that the underlying {{org.apache.hadoop.fs.azure.NativeAzureFileSystem}} offers, so I cannot comment on the correctness of the result. was (Author: kkl0u): What [~trohrmann] says is correct and it seems that this is the case since the introduction of the AzureFS support in Flink. So the AzureFS was never supported by the {{StreamingFileSink}}. For the {{StreamingFileSink}} to work with the {{HadoopRecoverableWriter}}, the writer should be in the classpath and the check [here|https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L60] should be changed to also accept {{wasb}}. That said, I am not sure about the guarantees (e.g. consistency) that the underlying {{org.apache.hadoop.fs.azure.NativeAzureFileSystem}} offers, so I cannot comment on the correctness of the result. > StreamingFileSink Azure HadoopRecoverableWriter class missing. > -------------------------------------------------------------- > > Key: FLINK-17444 > URL: https://issues.apache.org/jira/browse/FLINK-17444 > Project: Flink > Issue Type: Bug > Components: FileSystems > Affects Versions: 1.10.0 > Reporter: Marie May > Priority: Critical > Fix For: 1.11.0 > > > Hello, I was recently attempting to use the Streaming File Sink to store data > to Azure and get an exception error that it is missing the > HadoopRecoverableWriter. When I searched if anyone else had the issue I came > across this post here on [Stack Overflow > |[https://stackoverflow.com/questions/61246683/flink-streamingfilesink-on-azure-blob-storage]]. > Seeing no one responded, I asked about it on the mailing list and was told > to submit the issue here. > This is exception message they posted below but the stack overflow post goes > into more details of where they believe the issue comes from. > {code:java} > java.lang.NoClassDefFoundError: > org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter > > at > org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:202) > > at > org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.createRecoverableWriter(PluginFileSystemFactory.java:129) > > at > org.apache.flink.core.fs.SafetyNetWrapperFileSystem.createRecoverableWriter(SafetyNetWrapperFileSystem.java:69) > > at > org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.<init>(Buckets.java:117) > > at > org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:288) > > at > org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:402) > > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) > > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) > > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) > > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:284) > > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1006) > > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454) > > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449) > > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461) > > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) > > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)