[
https://issues.apache.org/jira/browse/FLINK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742615#comment-14742615
]
ASF GitHub Bot commented on FLINK-2583:
---------------------------------------
Github user aminouvic commented on the pull request:
https://github.com/apache/flink/pull/1084#issuecomment-139911603
Since the RollingSink was a little inspired by flume's HDFS Sink, it would
be nice to include another really valuable features that could make it more
complete.
One of the most common use cases of the HDFS Sink is to dispatch data into
multiple directories depending of attributes present in source events.
For example, let's say we have some data that have a timestamp and status
fields, one can specify in Flume conf file to write data into different
directories like this:
hadfs.path=/somepath/%{timestamp}/%{status}
The result will be to write data into multiple folders like
/somepath/some_timestamp/wellformed
/somepath/some_timestamp/malformed
/somepath/some_timestamp/incomplete
...
etc
To achieve this, Flume maintains a LRU Hashmap to hold not one but a set of
Writers, and computes the destination path for each event.
It also uses some params like maxOpenfFiles, idleTimeOut(optional, used to
close a file after x seconds of inactivity) to ensure not having too many open
files.
But to include this, the bucketing, cleanOnStartup and the checkpointing
logics need to be changed.
> Add Stream Sink For Rolling HDFS Files
> --------------------------------------
>
> Key: FLINK-2583
> URL: https://issues.apache.org/jira/browse/FLINK-2583
> Project: Flink
> Issue Type: New Feature
> Components: Streaming
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Fix For: 0.10
>
>
> In addition to having configurable file-rolling behavior the Sink should also
> integrate with checkpointing to make it possible to have exactly-once
> semantics throughout the topology.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
