[ https://issues.apache.org/jira/browse/HADOOP-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690828#comment-16690828 ]
wujinhu commented on HADOOP-15919: ---------------------------------- Thanks [~cheersyang] for your suggestions. I have fixed the issues you have mentioned. --------------------------------------------------------- {color:#f79232}+_TestOSSFileContext.java_+{color} {color:#f79232}+_This is an empty class._+{color} Yes, the class extends TestFileContext and will run tests in its parent. --------------------------------------------------------- {color:#f79232}+_AliyunOSSFileSystemStore_+{color} {color:#f79232}+_Not sure why you need to remove_+{color} {color:#f79232}+_statistics.incrementBytesWritten(file.length());_+{color} Because this patch added tests defined in FCStatisticsBaseTest, and I found this is a minor bug. So I fixed it and avoid duplicated calculation of bytesWritten metric. {code:java} diff --git a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java index 9c4435c11f3..3def6a7bed7 100644 --- a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java +++ b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java @@ -136,7 +136,7 @@ public FSDataOutputStream create(Path path, FsPermission permission, key, uploadPartSize, new SemaphoredDelegatingExecutor(boundedThreadPool, - blockOutputActiveBlocks, true)), (Statistics)(null)); + blockOutputActiveBlocks, true)), statistics); } LOG.debug(result.getETag()); statistics.incrementWriteOps(1); - statistics.incrementBytesWritten(file.length()); {code} Below is the unit test result in my mac env. {code:java} [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.688 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.027 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.464 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.308 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir [INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.214 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations [WARNING] Tests run: 68, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 74.481 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.018 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil [INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI [WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 27.65 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek [INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.293 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.672 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.827 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus [INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.349 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.18 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename [INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.013 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.597 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate [WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 9.673 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate [INFO] Running org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.472 s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen [INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.858 s - in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS [INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.462 s - in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.208 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 267.698 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream [INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.685 s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [INFO] [INFO] Results: [INFO] [WARNING] Tests run: 267, Failures: 0, Errors: 0, Skipped: 7 {code} > AliyunOSS: Enable Yarn to use OSS > --------------------------------- > > Key: HADOOP-15919 > URL: https://issues.apache.org/jira/browse/HADOOP-15919 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss > Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 > Reporter: wujinhu > Assignee: wujinhu > Priority: Major > Attachments: HADOOP-15919.001.patch, HADOOP-15919.002.patch, > HADOOP-15919.003.patch, HADOOP-15919.004.patch > > > Uses DelegateToFileSystem to expose AliyunOSSFileSystem as an > AbstractFileSystem -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org