[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732934#comment-15732934
]
ASF GitHub Bot commented on FLINK-5300:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/2971
[backport] [FLINK-5300] Add more gentle file deletion procedure
Backport of #2970 to the release-1.1 branch.
Before deleting a parent directory always check the directory whether it
contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not
instructed to
delete a non-empty directory.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink
backportMoreGentleFileDeletion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2971.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2971
----
commit fbce253eddd2ea6ce681b8881a5b6b8d470d861b
Author: Till Rohrmann <trohrm...@apache.org>
Date: 2016-12-08T17:53:40Z
[FLINK-5300] Add more gentle file deletion procedure
Before deleting a parent directory always check the directory whether it
contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not
instructed to
delete a non-empty directory.
----
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -------------------------------------------------------------------------------------------------
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Affects Versions: 1.2.0, 1.1.3
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
