[
https://issues.apache.org/jira/browse/HADOOP-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690828#comment-16690828
]
wujinhu commented on HADOOP-15919:
----------------------------------
Thanks [~cheersyang] for your suggestions. I have fixed the issues you have
mentioned.
---------------------------------------------------------
{color:#f79232}+_TestOSSFileContext.java_+{color}
{color:#f79232}+_This is an empty class._+{color}
Yes, the class extends TestFileContext and will run tests in its parent.
---------------------------------------------------------
{color:#f79232}+_AliyunOSSFileSystemStore_+{color}
{color:#f79232}+_Not sure why you need to remove_+{color}
{color:#f79232}+_statistics.incrementBytesWritten(file.length());_+{color}
Because this patch added tests defined in FCStatisticsBaseTest, and I found
this is a minor bug. So I fixed it and avoid duplicated calculation of
bytesWritten metric.
{code:java}
diff --git
a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
index 9c4435c11f3..3def6a7bed7 100644
---
a/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
+++
b/hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java
@@ -136,7 +136,7 @@ public FSDataOutputStream create(Path path, FsPermission
permission,
key,
uploadPartSize,
new SemaphoredDelegatingExecutor(boundedThreadPool,
- blockOutputActiveBlocks, true)), (Statistics)(null));
+ blockOutputActiveBlocks, true)), statistics);
}
LOG.debug(result.getETag());
statistics.incrementWriteOps(1);
- statistics.incrementBytesWritten(file.length());
{code}
Below is the unit test result in my mac env.
{code:java}
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.688 s
- in org.apache.hadoop.fs.aliyun.oss.TestAliyunCredentials
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.027 s
- in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSInputStream
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.464 s
- in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContext
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.308 s
- in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextStatistics
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.214
s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextCreateMkdir
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations
[WARNING] Tests run: 68, Failures: 0, Errors: 0, Skipped: 4, Time elapsed:
74.481 s - in
org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextMainOperations
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.018 s
- in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextUtil
[INFO] Running org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI
[WARNING] Tests run: 17, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
27.65 s - in org.apache.hadoop.fs.aliyun.oss.fileContext.TestOSSFileContextURI
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.293
s - in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractSeek
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.672 s
- in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractMkdir
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.827 s
- in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDelete
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus
[INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.349
s - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractGetFileStatus
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 67.18 s
- in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractDistCp
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
[INFO] Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.013 s
- in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRename
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.597 s
- in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractRootDir
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
[WARNING] Tests run: 11, Failures: 0, Errors: 0, Skipped: 2, Time elapsed:
9.673 s - in
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractCreate
[INFO] Running
org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.472 s
- in org.apache.hadoop.fs.aliyun.oss.contract.TestAliyunOSSContractOpen
[INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.858 s
- in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSS
[INFO] Running org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.462 s
- in org.apache.hadoop.fs.aliyun.oss.yarn.TestOSSMiniYarnCluster
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.208 s
- in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemStore
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 267.698
s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSBlockOutputStream
[INFO] Running org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[INFO] Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 39.685
s - in org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract
[INFO]
[INFO] Results:
[INFO]
[WARNING] Tests run: 267, Failures: 0, Errors: 0, Skipped: 7
{code}
> AliyunOSS: Enable Yarn to use OSS
> ---------------------------------
>
> Key: HADOOP-15919
> URL: https://issues.apache.org/jira/browse/HADOOP-15919
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/oss
> Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3
> Reporter: wujinhu
> Assignee: wujinhu
> Priority: Major
> Attachments: HADOOP-15919.001.patch, HADOOP-15919.002.patch,
> HADOOP-15919.003.patch, HADOOP-15919.004.patch
>
>
> Uses DelegateToFileSystem to expose AliyunOSSFileSystem as an
> AbstractFileSystem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
