zhangxinyu created FLINK-9367:
---------------------------------
Summary: Truncate() in BucketingSink is only allowed after
hadoop2.7
Key: FLINK-9367
URL: https://issues.apache.org/jira/browse/FLINK-9367
Project: Flink
Issue Type: Improvement
Components: Streaming Connectors
Affects Versions: 1.5.0
Reporter: zhangxinyu
Fix For: 1.5.0
When output to HDFS using BucketingSink, truncate() is only allowed after
hadoop2.7.
If some tasks failed, the ".valid-length" file is created for the lower version
hadoop.
The problem is, if other people want to use the data in HDFS, they must know
how to deal with the ".valid-length" file, otherwise, the data may be not
exactly-once.
I think it's not convenient for other people to use the data. Why not just read
the in-progress file and write a new file when restoring instead of writing a
".valid-length" file.
In this way, others who use the data in HDFS don't need to know how to deal
with the ".valid-length" file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
