Ted Malaska created HADOOP-10560: ------------------------------------ Summary: Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads Key: HADOOP-10560 URL: https://issues.apache.org/jira/browse/HADOOP-10560 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Ted Malaska Priority: Minor
In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3. do { PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true); for (FileMetadata file : listing.getFiles()) { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); } priorLastKey = listing.getPriorLastKey(); } while (priorLastKey != null); -- This message was sent by Atlassian JIRA (v6.2#6252)