[
https://issues.apache.org/jira/browse/FLINK-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363950#comment-16363950
]
Fabian Hueske commented on FLINK-3655:
--------------------------------------
Hi [~sjwiesman],
I've reworked the PR and opened a new one:
https://github.com/apache/flink/pull/5415
Unfortunately, we cannot magically enable this feature for all input formats
that are based on {{FileInputFormat}}, because it is a {{@Public}} interface.
With the changes that I proposed in the PR, we enable multipath support for the
CsvInputFormats, AvroInputFormat, OrcRowInputFormat, and TextInputFormat. All
other classes would have to override the {{supportsMultiPaths()}} method.
Can you check if the changes in the PR would address your use case?
It would be great if you could provide feedback soon because the feature freeze
for Flink 1.5.0 will happen in a few days.
Thank you, Fabian
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat
> ---------------------------------------------------------------------------------
>
> Key: FLINK-3655
> URL: https://issues.apache.org/jira/browse/FLINK-3655
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.0
> Reporter: Gna Phetsarath
> Assignee: Fabian Hueske
> Priority: Major
> Labels: starter
> Fix For: 1.5.0
>
>
> Allow comma-separated or multiple directories to be specified for
> FileInputFormat so that a DataSource will process the directories
> sequentially.
>
> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,/data/2016/01/03/*/*")
> in Scala
> env.readFile(paths: Seq[String])
> or
> env.readFile(path: String, otherPaths: String*)
> Wildcard support would be a bonus.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
