Github user aminouvic commented on the pull request:
https://github.com/apache/flink/pull/1084#issuecomment-139911603
Since the RollingSink was a little inspired by flume's HDFS Sink, it would
be nice to include another really valuable features that could make it more
complete.
One of the most common use cases of the HDFS Sink is to dispatch data into
multiple directories depending of attributes present in source events.
For example, let's say we have some data that have a timestamp and status
fields, one can specify in Flume conf file to write data into different
directories like this:
hadfs.path=/somepath/%{timestamp}/%{status}
The result will be to write data into multiple folders like
/somepath/some_timestamp/wellformed
/somepath/some_timestamp/malformed
/somepath/some_timestamp/incomplete
...
etc
To achieve this, Flume maintains a LRU Hashmap to hold not one but a set of
Writers, and computes the destination path for each event.
It also uses some params like maxOpenfFiles, idleTimeOut(optional, used to
close a file after x seconds of inactivity) to ensure not having too many open
files.
But to include this, the bucketing, cleanOnStartup and the checkpointing
logics need to be changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
