Github user kl0u commented on the issue:
https://github.com/apache/flink/pull/2797
The subtask matters for files that were opened but they did not make it
into a checkpoint because
a failure occurred after they were opened but before a checkpoint barrier
arrived. These are the ones
that are not cleaned up anymore. The rest, that are part of the checkpoint,
they are "cleaned upâ or
rolled back if needed.
Probably you may understand more how the cleaning up was happening if you
checkout the code
in the master for the cleanup.
> On Nov 15, 2016, at 10:12 PM, zentol <[email protected]> wrote:
>
> @zentol commented on this pull request.
>
> In
flink-streaming-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java
<https://github.com/apache/flink/pull/2797>:
>
> > }
>
> - @Override
> - public void restoreState(State<T> state) {
> - this.state = state;
> -
> - try {
> - initFileSystem();
> - } catch (IOException e) {
> - LOG.error("Error while creating FileSystem in
checkpoint restore.", e);
> - throw new RuntimeException("Error while creating
FileSystem in checkpoint restore.", e);
> - }
> + private void handleRestoredBucketState(BucketState<T> bucketState) {
> + // we can clean all the pending files since they were renamed to
> I don't understand why the subtask index even matters. In a simplified
view, every subtask maintains a set of Files. When rescaling/restoring state,
this set of files is given to another subtask. If this set contains a file
marked as pending, it should be allowed to delete it, since no other task
should have the same file since the state containing the information about the
file (aka, the file path) is not given to multiple subtasks.
>
> â
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
<https://github.com/apache/flink/pull/2797>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ACS1qP4VEEL7BZsKH1rlECWjHcQutwANks5q-iBSgaJpZM4KxHhq>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
