Yaroslav Tkachenko created FLINK-31285:
------------------------------------------
Summary: FileSource should support reading files in order
Key: FLINK-31285
URL: https://issues.apache.org/jira/browse/FLINK-31285
Project: Flink
Issue Type: New Feature
Components: Connectors / FileSystem
Affects Versions: 1.18.0
Reporter: Yaroslav Tkachenko
Currently, Flink's *FileSource* uses *LocalityAwareSplitAssigner* as a default
*FileSplitAssigner* and it doesn't guarantee any order. In many scenarios
involving processing historical data, reading files in order can be a
requirement, especially when using event-time processing.
I believe a new FileSplitAssigner should be implemented that supports ordering.
FileSourceBuilder should be extended to allow choosing a different
FileSplitAssigner.
It's also clear that the files may not be read in _perfect_ order with
parallelism > 1. However, in some cases, using parallelism of 1 might be fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
