[
https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987593#comment-13987593
]
Ted Malaska commented on HADOOP-10560:
--------------------------------------
Hey Steve,
Thank you for the review. Response by number below
1. Yes this is great. Do you want it in the same class or is there a Const
class that should house this
2. Agreed
3. Agreed
4. Agreed, Will do
5. I'm not sure how to do this one in a predictable way
6. Agreed
7. Agreed
8. I will check if that is supported with the current implementation. If not
I'll leave that change for another Jira
Regarding Jira 10400: Yes I've very aware of that Jira and I hope it continues
in its testing and gets committed. This Jira should not impact that jira in
any way. Now this jira benefits may be short lived, because S3A maybe better
in the long run. But I have short term needs for people using this File System
implementation today and who have tested this FileSystem implementation heavily.
Regarding the FindBugs issue: I am also updating the code to remove the find
bugs issue
I should have a new patch soon. Thanks again for the review
> Update NativeS3FileSystem to issue copy commands for files with in a
> directory with a configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Ted Malaska
> Assignee: Ted Malaska
> Priority: Minor
> Labels: performance
> Attachments: HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the
> files to the new location, but it will do this with one thread. Code is
> below. This jira will allow a configurable number of threads to be used to
> issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH,
> priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey +
> file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)
