[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732927#comment-15732927 ] ASF GitHub Bot commented on FLINK-5300: --- GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/2970 [FLINK-5300] Add more gentle file deletion procedure Before deleting a parent directory always check the directory whether it contains some files. If not, then try to delete the parent directory. This will give a more gentle behaviour wrt storage systems which are not instructed to delete a non-empty directory. cc: @StefanRRichter You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink moreGentleFileDeletion Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2970.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2970 commit 62d3d28f675aaad203d655848d30e9fb916af43b Author: Till Rohrmann Date: 2016-12-08T17:53:40Z [FLINK-5300] Add more gentle file deletion procedure Before deleting a parent directory always check the directory whether it contains some files. If not, then try to delete the parent directory. This will give a more gentle behaviour wrt storage systems which are not instructed to delete a non-empty directory. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732934#comment-15732934 ] ASF GitHub Bot commented on FLINK-5300: --- GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/2971 [backport] [FLINK-5300] Add more gentle file deletion procedure Backport of #2970 to the release-1.1 branch. Before deleting a parent directory always check the directory whether it contains some files. If not, then try to delete the parent directory. This will give a more gentle behaviour wrt storage systems which are not instructed to delete a non-empty directory. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink backportMoreGentleFileDeletion Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2971.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2971 commit fbce253eddd2ea6ce681b8881a5b6b8d470d861b Author: Till Rohrmann Date: 2016-12-08T17:53:40Z [FLINK-5300] Add more gentle file deletion procedure Before deleting a parent directory always check the directory whether it contains some files. If not, then try to delete the parent directory. This will give a more gentle behaviour wrt storage systems which are not instructed to delete a non-empty directory. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735088#comment-15735088 ] ASF GitHub Bot commented on FLINK-5300: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2970 I like the idea. I am wondering how expensive getting the array of `FileStatus` for all files in the directory is. HDFS in Hadoop 2 has the option to get a `ContentSummary` that has the number of files in a directory. I assume that this is more lightweight. We could extend Flink's FileSystem class to also offer something like that and then use that method. If we decide to not do that, it would be good to put the repeated logic for "delete if empty" into a utility function. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735089#comment-15735089 ] ASF GitHub Bot commented on FLINK-5300: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2971 I left a review on #2970 https://github.com/apache/flink/pull/2970#issuecomment-265994539 that can be applied here as well. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735472#comment-15735472 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2970 I like the idea of not listing the status for all contained files. However, I've looked at the implementation of Hadoop's `FileSystem#getContentSummary` and `FileSystem#listLocatedStatus` and both implementations call `FileSystem#listStatus`. Thus, unless this changes in the future, we wouldn't win a lot by calling the `getContentSummary` instead (actually we would have the overhead of aggregating the different `FileStatus` objects). Therefore, I'll refactor the code and add a `FileUtils#deleteDirectoryIfEmpty` method which will encapsulate the logic. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735539#comment-15735539 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2970 I've update this PR @StephanEwen. Unfortunately, I couldn't use Hadoop's `FileSystem#getContentSummary` because it will first request the status for the given path, then list all files and directories if the path is a directory. For each file it will aggregate the `FileStatus` and then recursively descend into each directory. Thus, I think that this method is not faster. I've refactored the code to contain a method `FileUtils#deletePathIfEmpty` to delete the path if it does not contain any files/directories. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735586#comment-15735586 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2971 I updated the PR wrt the results of the discussion in #2970. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735607#comment-15735607 ] ASF GitHub Bot commented on FLINK-5300: --- Github user uce commented on the issue: https://github.com/apache/flink/pull/2971 Looks good to be merged imo. I just kicked off RC2... If you would like to have this in 1.1.4, I can re-trigger the build. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735612#comment-15735612 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2971 Thanks for the review @uce. I think we can include this also in a possible 1.1.5 later. Who knows which other issues will still come up. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741632#comment-15741632 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2971 Failing test cases are unrelated. Merging this PR. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741644#comment-15741644 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2970 Rebasing on the latest master. @StephanEwen since I couldn't find a more efficient way to list the directory contents (wrt Hadoop FS) than `listStatus`, I think we can merge this PR. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741665#comment-15741665 ] ASF GitHub Bot commented on FLINK-5300: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2970 Looks good to me. I would actually suggest to add two tests, one in `flink-core` based on the local file system, and one in `flink-fs-tests`, based on HDFS. That way we make sure that there are no "unexpected behaviors", like some default file status always included (`.` or `..` or whatever). > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741946#comment-15741946 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2970 True. Will add the tests and then merge the PR. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745475#comment-15745475 ] ASF GitHub Bot commented on FLINK-5300: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2970 > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745477#comment-15745477 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/2971 Has been merged into the release-1.1 branch. > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[ https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745478#comment-15745478 ] ASF GitHub Bot commented on FLINK-5300: --- Github user tillrohrmann closed the pull request at: https://github.com/apache/flink/pull/2971 > FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete > non-empty directory > - > > Key: FLINK-5300 > URL: https://issues.apache.org/jira/browse/FLINK-5300 > Project: Flink > Issue Type: Improvement > Components: State Backends, Checkpointing >Affects Versions: 1.2.0, 1.1.3 >Reporter: Till Rohrmann >Assignee: Till Rohrmann >Priority: Critical > > Flink's behaviour to delete {{FileStateHandles}} and closing > {{FsCheckpointStateOutputStream}} always triggers a delete operation on the > parent directory. Often this call will fail because the directory still > contains some other files. > A user reported that the SRE of their Hadoop cluster noticed this behaviour > in the logs. It might be more system friendly if we first checked whether the > directory is empty or not. This would prevent many error message to appear in > the Hadoop logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)