[ https://issues.apache.org/jira/browse/FLINK-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Artsem Semianenka updated FLINK-10203: -------------------------------------- Summary: HadoopRecoverableFsDataOutputStream does not support Hadoop 2.6 (was: HadoopRecoverableWriter does not support Hadoop ) > HadoopRecoverableFsDataOutputStream does not support Hadoop 2.6 > --------------------------------------------------------------- > > Key: FLINK-10203 > URL: https://issues.apache.org/jira/browse/FLINK-10203 > Project: Flink > Issue Type: Bug > Components: DataStream API, filesystem-connector > Affects Versions: 1.6.0, 1.6.1, 1.7.0 > Reporter: Artsem Semianenka > Priority: Major > > New StreamingFileSink ( introduced in 1.6 Flink version ) use > HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS. > HadoopRecoverableFsDataOutputStream is an wrapper on FSDataOutputStream to > have an ability to restore from certain point of file after failure and > continue write data. To achive this recover functionality the > HadoopRecoverableFsDataOutputStream use "truncate" method which was > introduced only in Hadoop 2.7 . > Unfortently there are a few official Hadoop distibutives which latast version > still use Hadoop 2.6 (This distibutives: Cloudera, Pivotal HD ). As the > result Flink can't work with this distibutives. > Flink declares that supported Hadoop from version 2.4.0 upwards > (https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions) > I guess we should emulate the functionality of "truncate" method for older > Hadoop versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)