[
https://issues.apache.org/jira/browse/FLINK-2583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728744#comment-14728744
]
ASF GitHub Bot commented on FLINK-2583:
---------------------------------------
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/1084#issuecomment-137390414
I'm currently trying out the module. Some comments:
- Why do we name the module `flink-connector-hdfs`. I think a name such as
`flink-connector-filesystems` or `flink-connector-hfs` would be more
appropriate because its is implemented against Hadoops `FileSystems` classes,
and not HDFS classes.
So users should be able to use the connector with other FS such as Tachyon,
NFS, S3 etc.
- Is there a way of re-using existing InputFormats with the rolling file
sink? I guess users will start asking about CSV, Avro, Parquet ...
- I think there is already some code to monitor a file system directory to
ingest it into a data stream. Maybe it would make sense to move that code out
of the core into this module?
> Add Stream Sink For Rolling HDFS Files
> --------------------------------------
>
> Key: FLINK-2583
> URL: https://issues.apache.org/jira/browse/FLINK-2583
> Project: Flink
> Issue Type: New Feature
> Components: Streaming
> Reporter: Aljoscha Krettek
> Assignee: Aljoscha Krettek
> Fix For: 0.10
>
>
> In addition to having configurable file-rolling behavior the Sink should also
> integrate with checkpointing to make it possible to have exactly-once
> semantics throughout the topology.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
