[jira] [Commented] (HADOOP-18544) S3A: add option to disable probe for dir marker recreation on delete/rename.
[
https://issues.apache.org/jira/browse/HADOOP-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869380#comment-17869380
]
ASF GitHub Bot commented on HADOOP-18544:
-
steveloughran commented on PR #5354:
URL: https://github.com/apache/hadoop/pull/5354#issuecomment-2256199776
harshit, now that #6789 is in, we support an option
`fs.s3a.performance.flags` for a CSV list of flags. You could make this one of
them and people can enable it for applications which are ok with the changed
semantics
> S3A: add option to disable probe for dir marker recreation on delete/rename.
>
>
> Key: HADOOP-18544
> URL: https://issues.apache.org/jira/browse/HADOOP-18544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
> Labels: pull-request-available
>
> In applications which do many single-file deletions on the same dir, a lot of
> time is wasted in {{maybeCreateFakeParentDirectory()}}.
> Proposed: add an option to disable the probe, for use by applications which
> are happy for parent dirs to sometimes disappear after a cleanup.
> file by file delete is still woefully inefficient because of the HEAD request
> on every file, but there's no need to amplify the damage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18544) S3A: add option to disable probe for dir marker recreation on delete/rename.
[
https://issues.apache.org/jira/browse/HADOOP-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684678#comment-17684678
]
ASF GitHub Bot commented on HADOOP-18544:
-
hadoop-yetus commented on PR #5354:
URL: https://github.com/apache/hadoop/pull/5354#issuecomment-1419083013
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 53s | | trunk passed |
| +1 :green_heart: | compile | 23m 1s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 23s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 44s | | trunk passed |
| -1 :x: | javadoc | 1m 15s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 49s | | the patch passed |
| +1 :green_heart: | compile | 22m 26s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 26s | | the patch passed |
| +1 :green_heart: | compile | 20m 29s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 37s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 40 new + 49 unchanged - 0 fixed = 89 total (was
49) |
| +1 :green_heart: | mvnsite | 2m 38s | | the patch passed |
| -1 :x: | javadoc | 1m 6s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 18m 21s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 50s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 237m 26s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.fs.TestFilterFs |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.42 ServerAPI=1.42 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5354/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.c
[jira] [Commented] (HADOOP-18544) S3A: add option to disable probe for dir marker recreation on delete/rename.
[
https://issues.apache.org/jira/browse/HADOOP-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684537#comment-17684537
]
ASF GitHub Bot commented on HADOOP-18544:
-
HarshitGupta11 opened a new pull request, #5354:
URL: https://github.com/apache/hadoop/pull/5354
### Description of PR
In applications which do many single-file deletions on the same dir, a lot
of time is wasted in maybeCreateFakeParentDirectory().
Proposed: add an option to disable the probe, for use by applications which
are happy for parent dirs to sometimes disappear after a cleanup.
file by file delete is still woefully inefficient because of the HEAD
request on every file, but there's no need to amplify the damage.
### How was this patch tested?
The patch was tested against s3 bucket in US-West 2
### For code changes:
##Caveats:
Parent directories might disappear on delete or on renames.
##What breaks:
The rename tests are failing for the FileContext renames as both
S3AFileSystem and the FileContext have different probes and different rules.
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> S3A: add option to disable probe for dir marker recreation on delete/rename.
>
>
> Key: HADOOP-18544
> URL: https://issues.apache.org/jira/browse/HADOOP-18544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
>
> In applications which do many single-file deletions on the same dir, a lot of
> time is wasted in {{maybeCreateFakeParentDirectory()}}.
> Proposed: add an option to disable the probe, for use by applications which
> are happy for parent dirs to sometimes disappear after a cleanup.
> file by file delete is still woefully inefficient because of the HEAD request
> on every file, but there's no need to amplify the damage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18544) S3A: add option to disable probe for dir marker recreation on delete/rename.
[
https://issues.apache.org/jira/browse/HADOOP-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640168#comment-17640168
]
Steve Loughran commented on HADOOP-18544:
-
[~harshit.gupta] assigning to you
this'll need a new s3a option (Constants.java) read to a field in s3afs.
initialize(), then checked in {{maybeCreateFakeParentDirectory()}} to skip the
delete.
then need a test (similar to ITestS3ARenameCost/ITestS3ADeleteCost) which
asserts that no HEAD request is made on rename and delete. those existing
suites will need to set the new option to false to stop all their existing
tests failing; see their parent class's createConfiguration() to see what to do
there.
> S3A: add option to disable probe for dir marker recreation on delete/rename.
>
>
> Key: HADOOP-18544
> URL: https://issues.apache.org/jira/browse/HADOOP-18544
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.4
>Reporter: Steve Loughran
>Assignee: Harshit Gupta
>Priority: Major
>
> In applications which do many single-file deletions on the same dir, a lot of
> time is wasted in {{maybeCreateFakeParentDirectory()}}.
> Proposed: add an option to disable the probe, for use by applications which
> are happy for parent dirs to sometimes disappear after a cleanup.
> file by file delete is still woefully inefficient because of the HEAD request
> on every file, but there's no need to amplify the damage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
