GitHub user kl0u opened a pull request:
https://github.com/apache/flink/pull/1984
[FLINK-3889] Make File Monitoring Function checkpointable.
This pull request introduces the underlying functionality to make Streaming
File Sources persistent.
It does not yet change the API calls, as this will be done after agreeing
on the current architecture and
implementation.
In addition, this PR includes a commit for FLINK-3896. This allows an
operator to cancel its container task. The need for this functionality came
during a discussion with @StephanEwen and @aljoscha and it is a separate commit.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kl0u/flink ft_files
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1984.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1984
----
commit 7deb92236cec47ddcfbb3abfa396fd9d15f770b7
Author: kl0u <[email protected]>
Date: 2016-05-10T16:56:58Z
[FLINK-3896] Allow a StreamTask to be Externally Cancelled
It adds a method failExternally() to the StreamTask, so that custom
Operators
can make their containing task fail when needed.
commit c9682b7606451c4eecf6f2f6df9a498fb6d39577
Author: kl0u <[email protected]>
Date: 2016-04-10T14:56:42Z
[FLINK-3717] Make FileInputFormat checkpointable
This adds a new interface called CheckpointableInputFormat
which describes input formats whose state is queryable,
i.e. getCurrentChannelState() returns where the reader is
in the underlying source, and they can resume reading from
a user-specified position.
This functionality is not yet leveraged by current readers.
commit cbbfd8d7e6db0f8f114675b4047aecb94996e500
Author: kl0u <[email protected]>
Date: 2016-04-18T14:37:54Z
[FLINK-3889][FLINK-3808] Refactor File Monitoring Source
This is meant to replace the different file
reading sources in Flink streaming. Now there is
one monitoring source with DOP 1 monitoring a
directory and assigning input split to downstream
readers.
In addition, it makes the new features added by
FLINK-3717 and FLINK-3808 work together. Now we have
a file monitoring source that is also fault tolerant
and can guarantee exactly once semantics.
This does not replace the old API calls. This
will be done in a future commit.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
