Arvid Heise created FLINK-15777:
-----------------------------------
Summary: Truncate not working with plugins (as expected)
Key: FLINK-15777
URL: https://issues.apache.org/jira/browse/FLINK-15777
Project: Flink
Issue Type: Bug
Components: FileSystems
Affects Versions: 1.10.0, 1.11.0
Reporter: Arvid Heise
FLINK-14170 introduced Hadoop version check to support older versions with
limited functionality. However, the fix checks the Hadoop version of the system
classloader and not of the plugin (of the actual used filesystem).
The [version
checks|https://github.com/apache/flink/blob/a607bd9dbdda7d9925d6c351ba88d82edce2c571/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L197]
will fail when there is a Hadoop version in the classpath of Flink,
independent of the bundled version of the filesystem.
On a recent EMR setup with lingering Hadoop 2.6 libraries, this mean that you
cannot write to S3 with StreamingFileSink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
