GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/2763
[FLINK-5021] Makes the ContinuousFileReaderOperator rescalable.
This is the last PR that completes the refactoring of the
`ContinuousFileReaderOperator` so that it can be rescalable. With this, the
reader can restart from a savepoint with a different parallelism without
compromising the provided exactly-once guarantees.
The whole PR contains 3 commits.
The first removes the EOS special split which was used to signal that no
new splits are to be processed. This was useful in the `PROCESS_ONCE` mode. Now
the reader closes by setting a flag and waiting for all the pending splits to
be fully processed.
The second puts an additional check in the
`ContinuousFileMonitoringFunction` that guarantees that in the case of the
`PROCESS_ONCE`, the source will not reprocess the directory after recovering
from a failure.
Finally, the third integrates the new rescalable state abstractions with
the reader so that the reader can restart from a savepoint with different
parallelism and still guarantee exactly-once semantics.
R: @aljoscha
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink repart_fs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2763.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2763
----
commit 3976c70449f958da24ded2f7e5c3c1a3ef6cccd7
Author: kl0u <[email protected]>
Date: 2016-11-03T09:50:04Z
[FLINK-5021] Removes the special EOS TimestampedFileInputSplit.
Without this special split signaling that no more splits are
to arrive, the ContinuousFileReaderOperator now closes by
setting a flag that marks it as closed and exiting when the
flag is set to true and the pending split queue is empty.
commit 18ebac80f34c8c5db88d8493c21a3ba2e9fa2c6c
Author: kl0u <[email protected]>
Date: 2016-11-03T10:08:45Z
[FLINK-5021] Guarantees PROCESS_ONCE works correctly after recovering.
commit bbabe5e95aa6f115772099af250744a066eccb77
Author: kl0u <[email protected]>
Date: 2016-11-03T10:21:08Z
[FLINK-5021] Makes the ContinuousFileReaderOperator rescalable.
This is the last commit that completes the refactoring of the
ContinuousFileReaderOperator so that it can be rescalable.
With this, the reader can restart from a savepoint with a
different parallelism without compromising the provided
exactly-once guarantees.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
