[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923434#comment-16923434 ] Steve Loughran commented on HADOOP-16430: - committed -thanks for the review. Yes, I like a lot of the GH review mechanism, though I still also like the ability to do a quick summary. Sometimes I feel GH is more optimised for reviewing details than the overall patch. And once you start rebasing longer-lived patches, things get tricky. Have a play with the vs.code github integration if you haven't -you can review the patch with all the code locally checked out, adding comments in the IDE. This is very slick, especially for acting on people's suggestions. > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923431#comment-16923431 ] Hudson commented on HADOOP-16430: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17231 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17231/]) HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with (stevel: rev 511df1e837b19ccb9271520589452d82d50ac69d) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FutureIOSupport.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/ExecutingStoreOperation.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StoreContext.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMetadataPersistenceException.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3AFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/ITestPartialRenamesDeletes.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3ADeleteManyFiles.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/DeleteOperation.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Listing.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/InternalIterators.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DelayedUpdateRenameTracker.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AFailureHandling.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InternalConstants.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/select/InternalSelectConstants.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextURIBase.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/MultiObjectDeleteSupport.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ALocatedFileStatus.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/AbstractITestS3AMetadataStoreScale.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/InternalConstants.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractRootDir.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/OperationCallbacks.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardListConsistency.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RenameOperation.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/ProgressiveRenameTracker.j
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922903#comment-16922903 ] Aaron Fabbri commented on HADOOP-16430: --- Latest PR commits reviewed (+1). Aside: This is nice not having to run diff on a diff to just see what changed. Nice to be able to follow your commit history in the PR. > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918773#comment-16918773 ] Aaron Fabbri commented on HADOOP-16430: --- +1 LGTM after your clarifying comments.. Thanks for the contribution. > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918167#comment-16918167 ] Aaron Fabbri commented on HADOOP-16430: --- Thanks for the work on this. Looks pretty good. Made a couple of comments on the PR. > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914182#comment-16914182 ] Steve Loughran commented on HADOOP-16430: - Highlight: coding this has thrown up an issue with the current implementation. the recursive delete only asks S3 for files, not DDB. This ensures that incomplete DDB stores are not a problem, so that deletion does mostly recover from any problems. However, it relies on LIST being complete, which we know is untrue as it is eventually consistent. Proposed: after deleting all files, we do a recursive list of what is left in S3Guard and delete those too. That way files we know about but which the list missed will still get cleaned up. > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0, 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886461#comment-16886461 ] Steve Loughran commented on HADOOP-16430: - Attached: screenshot of on-demand DDB load during a test run. IOPs goes up to 300 for a read, write to ~250 but for slightly longer. DDB shows 3ms per operation. Delete doesn't do batches; it just goes through file-by-file. If we were confident that you did see a bulk write speedup (probably uses up fewer HTTPS connections at the very least), it could be considered/justlfied > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: Screenshot 2019-07-16 at 22.08.31.png > > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[
https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886460#comment-16886460
]
Steve Loughran commented on HADOOP-16430:
-
h3. s3guard + local + single file delete
{code}
Starting: Rename s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src
to s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 20:53:49,226 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
9:39.621s
2019-07-16 20:53:49,227 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(78)) - Effective rename
bandwidth 0.000146 MB/s
2019-07-16 20:57:31,942 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Delete
subtree s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 21:03:58,417 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
6:26.475s
2019-07-16 21:03:58,417 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(101)) - Timer per object
deletion 38.648592 milliseconds
2019-07-16 21:03:59,556 [teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
{code}
h3. s3guard + local + bulk delete
{code}
2019-07-16 21:09:12,297 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 21:12:54,386 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
3:42.089s
2019-07-16 21:12:54,386 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(78)) - Effective rename
bandwidth 0.000382 MB/s
2019-07-16 21:16:43,691 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Delete
subtree s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 21:17:19,875 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
0:36.183s
2019-07-16 21:17:19,875 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(101)) - Timer per object
deletion 3.618443 milliseconds
{code}
So again, 10x speedup in delete time during tree delete. For rename, the
differences are smaller; and even with nearly empty files, the overhead of the
copy overwhelms that of the delete.
If we really wanted to speed things up, we could try doing the single/bulk
delete call async, so that for each max-of-5K file list, we can spin off up to
5 delete calls and wait for them to finish. At the same time, this'd lead to a
massive spike in parallelised DDB writes; though as they'd go through the same
pool it'd be limited by pool capacity
> S3AFilesystem.delete to incrementally update s3guard with deletions
> ---
>
> Key: HADOOP-16430
> URL: https://issues.apache.org/jira/browse/HADOOP-16430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: Screenshot 2019-07-16 at 22.08.31.png
>
>
> Currently S3AFilesystem.delete() only updates the delete at the end of a
> paged delete operation. This makes it slow when there are many thousands of
> files to delete ,and increases the window of vulnerability to failures
> Preferred
> * after every bulk DELETE call is issued to S3, queue the (async) delete of
> all entries in that post.
> * at the end of the delete, await the completion of these operations.
> * inside S3AFS, also do the delete across threads, so that different HTTPS
> connections can be used.
> This should maximise DDB throughput against tables which aren't IO limited.
> When executed against small IOP limited tables, the parallel DDB DELETE
> batches will trigger a lot of throttling events; we should make
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[
https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886455#comment-16886455
]
Steve Loughran commented on HADOOP-16430:
-
For the curious
h3. single file delete
{code}
2019-07-16 17:40:27,487 [JUnit-testBulkRenameAndDelete] INFO
impl.ITestPartialRenamesDeletes (DurationInfo.java:(72)) - Starting:
Creating 1 files
2019-07-16 17:42:02,633 [JUnit-testBulkRenameAndDelete] INFO
impl.ITestPartialRenamesDeletes (DurationInfo.java:close(87)) - Creating 1
files: duration 1:35.146s
2019-07-16 17:45:01,905 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 17:54:40,158 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
9:38.253s
2019-07-16 17:54:40,162 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(78)) - Effective rename
bandwidth 0.000147 MB/s
2019-07-16 17:58:17,999 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Delete
subtree s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 18:04:49,074 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
6:31.075s
2019-07-16 18:04:49,074 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(101)) - Nanoseconds per
object deleted 39108 microseconds
2019-07-16 18:04:49,684 [teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
{code}
h3. Multifile delete
{code}
2019-07-16 18:04:50,625 [JUnit-testBulkRenameAndDelete] INFO
impl.ITestPartialRenamesDeletes (DurationInfo.java:(72)) - Starting:
Creating 1 files
2019-07-16 18:06:33,128 [JUnit-testBulkRenameAndDelete] INFO
impl.ITestPartialRenamesDeletes (DurationInfo.java:close(87)) - Creating 1
files: duration 1:42.503s
2019-07-16 18:09:35,521 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 18:13:12,657 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Rename
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/src to
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
3:37.136s
2019-07-16 18:13:12,658 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(78)) - Effective rename
bandwidth 0.000390 MB/s
2019-07-16 18:17:09,695 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:(72)) - Starting: Delete
subtree s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final
2019-07-16 18:17:42,863 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles (DurationInfo.java:close(87)) - Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
0:33.168s
2019-07-16 18:17:42,864 [JUnit-testBulkRenameAndDelete] INFO
scale.ITestS3ADeleteManyFiles
(ITestS3ADeleteManyFiles.java:testBulkRenameAndDelete(101)) - Nanoseconds per
object deleted 3316 microseconds
2019-07-16 18:17:43,588 [teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
{code}
The bulk rename took 9:38 on single delete, 3:37 on multidelete, ~3x longer.
On the delete alone, multifile delete: 0:33. Single delete, 6:31 -over 10x
slower.
At the same time *only* 10x slower, even though we are now pushing out 1000
entries to a delete in a single file.
> S3AFilesystem.delete to incrementally update s3guard with deletions
> ---
>
> Key: HADOOP-16430
> URL: https://issues.apache.org/jira/browse/HADOOP-16430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> Currently S3AFilesystem.delete() only updates the delete at the end of a
> paged delete operation. This makes it slow when there are many thousands of
> files to dele
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886243#comment-16886243 ] Steve Loughran commented on HADOOP-16430: - update: the patch. Use of keyToPath over keyToQualifiedPath > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[
https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886240#comment-16886240
]
Steve Loughran commented on HADOOP-16430:
-
Managed to create an NPE in test teardown here in a test run:
-Dit.test=ITestS3ADeleteManyFiles -Dscale -Ds3guard -Ddynamo
I'm assuming this is my new patch for now, but it could be a sign of something
which has been lurking for a bit and its only with one of us bumping up the
scale count it surfaces.
{code}
scale.test.operation.count
2000
{code}
{code}
Delete subtree
s3a://hwdev-steve-ireland-new/test/testBulkRenameAndDelete/final: duration
0:01.605s
2019-07-16 16:52:14,179 [teardown] ERROR contract.ContractTestUtils
(ContractTestUtils.java:cleanup(380)) - Error deleting in TEARDOWN - /test:
java.lang.NullPointerException: Path
/test/testBulkRenameAndDelete/final/file-1899 missing scheme
java.lang.NullPointerException: Path
/test/testBulkRenameAndDelete/final/file-1899 missing scheme
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:980)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.checkPath(DynamoDBMetadataStore.java:2092)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerDelete(DynamoDBMetadataStore.java:577)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$deleteEntries$2(DynamoDBMetadataStore.java:676)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:61)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-07-16 16:52:14,182 [teardown] INFO contract.AbstractFSContractTestBase
(AbstractFSContractTestBase.java:describe(255)) - closing file system
> S3AFilesystem.delete to incrementally update s3guard with deletions
> ---
>
> Key: HADOOP-16430
> URL: https://issues.apache.org/jira/browse/HADOOP-16430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Priority: Major
>
> Currently S3AFilesystem.delete() only updates the delete at the end of a
> paged delete operation. This makes it slow when there are many thousands of
> files to delete ,and increases the window of vulnerability to failures
> Preferred
> * after every bulk DELETE call is issued to S3, queue the (async) delete of
> all entries in that post.
> * at the end of the delete, await the completion of these operations.
> * inside S3AFS, also do the delete across threads, so that different HTTPS
> connections can be used.
> This should maximise DDB throughput against tables which aren't IO limited.
> When executed against small IOP limited tables, the parallel DDB DELETE
> batches will trigger a lot of throttling events; we should make sure these
> aren't going to trigger failures
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886155#comment-16886155 ] Steve Loughran commented on HADOOP-16430: - This is hard to do efficiently without that same tracking (or more) of the rename call. Specifically: currently a delete(path) will update the parent dir. But if we know the delete is taking place under a directory tree, then the only directories which we know are being deleted are those which are not under the destination directory. This leads to a design of # create ancestor state context with dest dir = base dir of delete # add a delete(list paths, state) operation and then for all entries in the path list * Put a tombstone if they are a file * Directory (how to check?): also put a tombstone * Only update the parent dir if the path being changed is that of the base directory of the delete * At the end of the delete, finish off by making sure there's a tombstone in each dir and no child which isn't. Like I warned: more complex. Not much smaller than that of rename. In fact, the operation is essentially that of Metastore.move() as used in the ProgressiveRenameTracker. > S3AFilesystem.delete to incrementally update s3guard with deletions > --- > > Key: HADOOP-16430 > URL: https://issues.apache.org/jira/browse/HADOOP-16430 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Priority: Major > > Currently S3AFilesystem.delete() only updates the delete at the end of a > paged delete operation. This makes it slow when there are many thousands of > files to delete ,and increases the window of vulnerability to failures > Preferred > * after every bulk DELETE call is issued to S3, queue the (async) delete of > all entries in that post. > * at the end of the delete, await the completion of these operations. > * inside S3AFS, also do the delete across threads, so that different HTTPS > connections can be used. > This should maximise DDB throughput against tables which aren't IO limited. > When executed against small IOP limited tables, the parallel DDB DELETE > batches will trigger a lot of throttling events; we should make sure these > aren't going to trigger failures -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
