[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732927#comment-15732927
]
ASF GitHub Bot commented on FLINK-5300:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/2970
[FLINK-5300] Add more gentle file deletion procedure
Before deleting a parent directory always check the directory whether it
contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not
instructed to
delete a non-empty directory.
cc: @StefanRRichter
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink moreGentleFileDeletion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2970.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2970
commit 62d3d28f675aaad203d655848d30e9fb916af43b
Author: Till Rohrmann
Date: 2016-12-08T17:53:40Z
[FLINK-5300] Add more gentle file deletion procedure
Before deleting a parent directory always check the directory whether it
contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not
instructed to
delete a non-empty directory.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732934#comment-15732934
]
ASF GitHub Bot commented on FLINK-5300:
---
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/2971
[backport] [FLINK-5300] Add more gentle file deletion procedure
Backport of #2970 to the release-1.1 branch.
Before deleting a parent directory always check the directory whether it
contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not
instructed to
delete a non-empty directory.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink
backportMoreGentleFileDeletion
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2971.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2971
commit fbce253eddd2ea6ce681b8881a5b6b8d470d861b
Author: Till Rohrmann
Date: 2016-12-08T17:53:40Z
[FLINK-5300] Add more gentle file deletion procedure
Before deleting a parent directory always check the directory whether it
contains some
files. If not, then try to delete the parent directory.
This will give a more gentle behaviour wrt storage systems which are not
instructed to
delete a non-empty directory.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735088#comment-15735088
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2970
I like the idea.
I am wondering how expensive getting the array of `FileStatus` for all
files in the directory is. HDFS in Hadoop 2 has the option to get a
`ContentSummary` that has the number of files in a directory. I assume that
this is more lightweight.
We could extend Flink's FileSystem class to also offer something like that
and then use that method.
If we decide to not do that, it would be good to put the repeated logic for
"delete if empty" into a utility function.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735089#comment-15735089
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2971
I left a review on #2970
https://github.com/apache/flink/pull/2970#issuecomment-265994539
that can be applied here as well.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735472#comment-15735472
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2970
I like the idea of not listing the status for all contained files. However,
I've looked at the implementation of Hadoop's `FileSystem#getContentSummary`
and `FileSystem#listLocatedStatus` and both implementations call
`FileSystem#listStatus`. Thus, unless this changes in the future, we wouldn't
win a lot by calling the `getContentSummary` instead (actually we would have
the overhead of aggregating the different `FileStatus` objects).
Therefore, I'll refactor the code and add a
`FileUtils#deleteDirectoryIfEmpty` method which will encapsulate the logic.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735539#comment-15735539
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2970
I've update this PR @StephanEwen. Unfortunately, I couldn't use Hadoop's
`FileSystem#getContentSummary` because it will first request the status for the
given path, then list all files and directories if the path is a directory. For
each file it will aggregate the `FileStatus` and then recursively descend into
each directory. Thus, I think that this method is not faster.
I've refactored the code to contain a method `FileUtils#deletePathIfEmpty`
to delete the path if it does not contain any files/directories.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735586#comment-15735586
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2971
I updated the PR wrt the results of the discussion in #2970.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735607#comment-15735607
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user uce commented on the issue:
https://github.com/apache/flink/pull/2971
Looks good to be merged imo. I just kicked off RC2... If you would like to
have this in 1.1.4, I can re-trigger the build.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735612#comment-15735612
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2971
Thanks for the review @uce. I think we can include this also in a possible
1.1.5 later. Who knows which other issues will still come up.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741632#comment-15741632
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2971
Failing test cases are unrelated. Merging this PR.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741644#comment-15741644
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2970
Rebasing on the latest master. @StephanEwen since I couldn't find a more
efficient way to list the directory contents (wrt Hadoop FS) than `listStatus`,
I think we can merge this PR.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741665#comment-15741665
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/2970
Looks good to me. I would actually suggest to add two tests, one in
`flink-core` based on the local file system, and one in `flink-fs-tests`, based
on HDFS.
That way we make sure that there are no "unexpected behaviors", like some
default file status always included (`.` or `..` or whatever).
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741946#comment-15741946
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2970
True. Will add the tests and then merge the PR.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745475#comment-15745475
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2970
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745477#comment-15745477
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/2971
Has been merged into the release-1.1 branch.
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-5300) FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete non-empty directory
[
https://issues.apache.org/jira/browse/FLINK-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15745478#comment-15745478
]
ASF GitHub Bot commented on FLINK-5300:
---
Github user tillrohrmann closed the pull request at:
https://github.com/apache/flink/pull/2971
> FileStateHandle#discard & FsCheckpointStateOutputStream#close tries to delete
> non-empty directory
> -
>
> Key: FLINK-5300
> URL: https://issues.apache.org/jira/browse/FLINK-5300
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
>
> Flink's behaviour to delete {{FileStateHandles}} and closing
> {{FsCheckpointStateOutputStream}} always triggers a delete operation on the
> parent directory. Often this call will fail because the directory still
> contains some other files.
> A user reported that the SRE of their Hadoop cluster noticed this behaviour
> in the logs. It might be more system friendly if we first checked whether the
> directory is empty or not. This would prevent many error message to appear in
> the Hadoop logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
