[
https://issues.apache.org/jira/browse/FLINK-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935904#comment-16935904
]
John Lonergan commented on FLINK-14170:
---------------------------------------
Yep it's unnecessarily restrictive and actually breaks Parquest even though it
would otherwise work just fine on Hadoop 2.6.
Remove the global check in construction and instead the make the code throw an
"NotImplementedException" +only+ if a sink actually happens to make that call.
> Support hadoop < 2.7 with StreamingFileSink.BulkFormatBuilder
> -------------------------------------------------------------
>
> Key: FLINK-14170
> URL: https://issues.apache.org/jira/browse/FLINK-14170
> Project: Flink
> Issue Type: Improvement
> Components: API / DataSet
> Affects Versions: 1.8.0, 1.8.1, 1.8.2, 1.9.0
> Reporter: Bhagavan
> Priority: Major
>
> Currently, StreamingFileSink is supported only with Hadoop >= 2.7
> irrespective of Row/bulk format builder. This restriction is due to truncate
> is not supported in Hadoop < 2.7
> However, BulkFormatBuilder does not use truncate method to restore the file.
> So the restricting StreamingFileSink.BulkFormatBuilder to be used only with
> Hadoop >= 2.7 is not necessary.
> So requested improvement is to remove the precondition on
> HadoopRecoverableWriter and allow BulkFormatBuilder (Parquet) to be used in
> Hadoop 2.6 ( Most of the enterprises still on CDH 5.x)
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
