[
https://issues.apache.org/jira/browse/FLINK-10203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590352#comment-16590352
]
ASF GitHub Bot commented on FLINK-10203:
----------------------------------------
StephanEwen commented on issue #6608: [FLINK-10203]Support truncate method for
old Hadoop versions in HadoopRecoverableFsDataOutputStream
URL: https://github.com/apache/flink/pull/6608#issuecomment-415451715
@art4ul Initially, we wanted to keep the new StreamingFileSink code simple
and clean, that's why we decided to only with Hadoop 2.7+ for HDFS and retained
the old BucketingSink, to support prior Hadoop versions.
Is it a critical issue on your side to not use the BucketingSink?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Support truncate method for old Hadoop versions in
> HadoopRecoverableFsDataOutputStream
> --------------------------------------------------------------------------------------
>
> Key: FLINK-10203
> URL: https://issues.apache.org/jira/browse/FLINK-10203
> Project: Flink
> Issue Type: Bug
> Components: DataStream API, filesystem-connector
> Affects Versions: 1.6.0, 1.6.1, 1.7.0
> Reporter: Artsem Semianenka
> Priority: Major
> Labels: pull-request-available
>
> New StreamingFileSink ( introduced in 1.6 Flink version ) use
> HadoopRecoverableFsDataOutputStream wrapper to write data in HDFS.
> HadoopRecoverableFsDataOutputStream is an wrapper on FSDataOutputStream to
> have an ability to restore from certain point of file after failure and
> continue write data. To achive this recover functionality the
> HadoopRecoverableFsDataOutputStream use "truncate" method which was
> introduced only in Hadoop 2.7 .
> Unfortently there are a few official Hadoop distibutives which latast version
> still use Hadoop 2.6 (This distibutives: Cloudera, Pivotal HD ). As the
> result Flink can't work with this distibutives.
> Flink declares that supported Hadoop from version 2.4.0 upwards
> (https://ci.apache.org/projects/flink/flink-docs-release-1.6/start/building.html#hadoop-versions)
> I guess we should emulate the functionality of "truncate" method for older
> Hadoop versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
