[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557769#comment-15557769 ] Genmao Yu edited comment on HADOOP-12756 at 10/8/16 11:38 AM: -- h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (!) |meaningless and no need| |chmod| (!) |meaningless and no need | |chown| (!) | meaningless and no need| |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |only support oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | was (Author: unclegen): h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) |meaningless and no need| |chmod| (x) |meaningless and no need | |chown| (x) | meaningless and no need| |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |only support oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration-v2.pdf, > Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HCFS User manual.md, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557769#comment-15557769 ] Genmao Yu edited comment on HADOOP-12756 at 10/8/16 11:37 AM: -- h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) |meaningless and no need| |chmod| (x) |meaningless and no need | |chown| (x) | meaningless and no need| |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |only support oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | was (Author: unclegen): h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) | | |chmod| (x) | | |chown| (x) | | |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |only support oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration-v2.pdf, > Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HCFS User manual.md, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557769#comment-15557769 ] Genmao Yu edited comment on HADOOP-12756 at 10/8/16 11:22 AM: -- h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) | | |chmod| (x) | | |chown| (x) | | |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |only support oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | was (Author: unclegen): h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) | | |chmod| (x) | | |chown| (x) | | |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration-v2.pdf, > Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HCFS User manual.md, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557769#comment-15557769 ] Genmao Yu edited comment on HADOOP-12756 at 10/8/16 11:22 AM: -- h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) | | |chmod| (x) | | |chown| (x) | | |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs, local -> oss, hdfs -> oss | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | was (Author: unclegen): h4. FS Shell ||op||support or not||comment|| |cat| (/) | | |chgrp| (x) | | |chmod| (x) | | |chown| (x) | | |copyFromLocal| (/) | | |copyToLocal| (/) | | |cp| (/) | oss -> oss, oss -> local, oss -> hdfs | |du| (/) | | |dus| (/) | | |expunge| (x) |only support hdfs | |get| (/) | | |getmerge| (/) | | |ls| (/) | | |lsr| (/) | | |mkdir| (/) | | |movefromLocal| (/) | | |mv| (/) |oss -> oss | |put| (/) | | |rm| (/) | | |rmr| (/) | | |setrep| (x) | locked | |stat| (/) | | |tail| (/) | | |test| (/) | | |text| (/) | | |touchz| (/) | | > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration-v2.pdf, > Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HCFS User manual.md, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527773#comment-15527773 ] Anu Engineer edited comment on HADOOP-12756 at 9/28/16 12:24 AM: - [~drankye] I think what Arpit is saying is that he does *not* have an issue with the code. The proper process to bring in this code would be to call for vote. Again, it is nothing to do with Aliyun code or technical issues. It gives the community a chance to review, understand and comment upon the code base before it is committed. That I think would be the best way to build a community of contributors around this feature. If you agree that we should follow the right process, I think we should *revert* this change and call for a merge vote and merge based on the results of such a voting thread. The danger of the precedent what we are doing in this branch would be that someone else might decide to bring in another feature via this loophole saying that this was done in Aliyun code merge. That is what I think we want to avoid, in many senses a rule of law remains a rule only if it is followed consistently. I am really sympathetic to what was done and I appreciate the enthusiasm and the spirit of let us get it done, but I think this list of changes is large enough for us to follow the right process. As far as I can see, few days spend on voting time will only strengthen the sense of community around this code base. [~andrew.wang] Since this is a single commit, reverting and merging will actuall be a better experience, because it will allow you follow the policy that was suggested by you "git merge --no-ff" is also the preferred way of integrating a feature branch to other branches, e.g. branch-2." >From >https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E was (Author: anu): [~drankye] I think what Arpit is saying is that he does *not* have an issue with the code. The proper process to bring in this code would be to call for vote. Again, it is nothing to do with Aliyun code or technical issues. It gives the community a chance to review, understand and comment upon the code base before it is committed. That I think would be the best way to build a community of contributors around this feature. If you agree that we should follow the right process, I think we should *revert* this change and call for a merge vote and merge based on the results of such a voting thread. The danger of the precedent what we are doing in this branch would be that someone else might decide to bring in another feature via this loophole saying that this was done in Aliyun code merge. That is what I think we want to avoid, in many senses a rule of law remains a rule only if it is followed consistently. I am really sympathetic to what was done and I appreciate the enthusiasm and the spirit of let us get it done, but I think this list of changes is large enough for us for us to follow the right process. As far as I can see, few days spend on voting time will only strengthen the sense of community around this code base. [~andrew.wang] Since this is a single commit, reverting and merging will actuall be a better experience, because it will allow you follow the policy that was suggested by you "git merge --no-ff" is also the preferred way of integrating a feature branch to other branches, e.g. branch-2." >From >https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration-v2.pdf, > Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HCFS User manual.md, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527530#comment-15527530 ] Kai Zheng edited comment on HADOOP-12756 at 9/27/16 9:55 PM: - Hi [~arpitagarwal], There wasn't an explicit vote thread called for this in the mailing list. I tracked the important discussions in this master issue, in [above comment|https://issues.apache.org/jira/browse/HADOOP-12756?focusedCommentId=15511801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15511801] made the summary about this branch work and called for the merge. I wish it could serve the same purpose and would work for you as well. The merge was recorded [here|https://issues.apache.org/jira/browse/HADOOP-12756?focusedCommentId=15520800=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15520800] and how would you like it? Thank you for the discussion. was (Author: drankye): Hi [~arpitagarwal], Yes there wasn't an explicit vote thread called for this in the mailing list. I tracked the important discussions in this master issue, in [above comment|https://issues.apache.org/jira/browse/HADOOP-12756?focusedCommentId=15511801=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15511801] made the summary about this branch work and called for the merge. I wish it could serve the same purpose and would work for you as well. The merge was recorded [here|https://issues.apache.org/jira/browse/HADOOP-12756?focusedCommentId=15520800=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15520800] and how would you like it? Thank you for the discussion. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration-v2.pdf, > Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, > HADOOP-12756.007.patch, HADOOP-12756.008.patch, HADOOP-12756.009.patch, > HADOOP-12756.010.patch, HCFS User manual.md, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506808#comment-15506808 ] Genmao Yu edited comment on HADOOP-12756 at 9/20/16 3:05 PM: - [~drankye] +1 to your suggestion, but the truth is many developers are familiar with ‘oss://’ in Aliyun E-MapReduce, and Aliyun OSS itself is using 'oss://' in many places, like https://help.aliyun.com/document_detail/32185.html. So, i think it is better to continue to use 'oss://'. was (Author: unclegen): [~drankye] +1 to your suggestion, but the truth is many developers are familiar with ‘oss’ in Aliyun E-MapReduce, and Aliyun OSS itself is using 'oss://' in many places, like https://help.aliyun.com/document_detail/32185.html. So, i think it is better to continue to use 'oss'. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0, HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, > HADOOP-12756.003.patch, HADOOP-12756.004.patch, HADOOP-12756.005.patch, > HADOOP-12756.006.patch, HADOOP-12756.007.patch, HADOOP-12756.008.patch, > HADOOP-12756.009.patch, HADOOP-12756.010.patch, HCFS User manual.md, OSS > integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445378#comment-15445378 ] shimingfei edited comment on HADOOP-12756 at 8/29/16 10:09 AM: --- [~drankye] [~steve_l] Thanks for your reviewing and comments. We are doing iterations on current code base, and will propose a proposal to merge to trunk. Current code uses FS contract test, and the test can be enabled/disabled by configurations, we are glad to help to set up the tests for Aliyun OSS, when the FS contract tests are changed was (Author: shimingfei): [~drankye] [~stevea] Thanks for your reviewing and comments. We are doing iterations on current code base, and will propose a proposal to merge to trunk. Current code uses FS contract test, and the test can be enabled/disabled by configurations, we are glad to help to set up the tests for Aliyun OSS, when the FS contract tests are changed > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0, HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: Aliyun-OSS-integration.pdf, HADOOP-12756-v02.patch, > HADOOP-12756.003.patch, HADOOP-12756.004.patch, HADOOP-12756.005.patch, > HADOOP-12756.006.patch, HADOOP-12756.007.patch, HADOOP-12756.008.patch, > HADOOP-12756.009.patch, HADOOP-12756.010.patch, HCFS User manual.md, OSS > integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383693#comment-15383693 ] uncleGen edited comment on HADOOP-12756 at 7/19/16 7:08 AM: [~shimingfei] IMHO, when do 'multipartUploadObject' operation in class 'AliyunOSSOutputStream', the part number is less than or equal to 1, so the part size need to be limited by 'fs.oss.multipart.upload.size' and part number upper limit (now is 1). See the doc [here|https://help.aliyun.com/document_detail/31993.html?spm=5176.product31815.6.265.iPB9WC]. was (Author: unclegen): [~shimingfei] IMHO, when do 'multipartUploadObject' operation in class 'AliyunOSSOutputStream', the part number is less than or equal to 1, so the part size need to be limited by 'fs.oss.multipart.upload.size' and part number upper limit (now is 1). See the doc [here](https://help.aliyun.com/document_detail/31993.html?spm=5176.product31815.6.265.iPB9WC). > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0, HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, HCFS > User manual.md, OSS integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381647#comment-15381647 ] uncleGen edited comment on HADOOP-12756 at 7/18/16 3:00 AM: Yep, we met the same problem and attempted to scratch the accessKeyId/accessKeySecret information, but indeed it did not work properly. For better security, we need to cut out the accessKeyId/accessKeySecret from OSS URI. was (Author: unclegen): Yep, we met the same problem and attempted to scratch the accessKeyId/accessKeySecret information, but indeed it did not work properly. And, I am not sure whether AliyunOSSFileSystem can get accessKeyId/accessKeySecret properly when set in config in code in some cases, like resource localization. But, for better security, we need to cut out the accessKeyId/accessKeySecret from OSS URI. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0, HADOOP-12756 >Reporter: shimingfei >Assignee: shimingfei > Fix For: HADOOP-12756 > > Attachments: HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HADOOP-12756.006.patch, HCFS > User manual.md, OSS integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15346018#comment-15346018 ] shimingfei edited comment on HADOOP-12756 at 6/23/16 8:12 AM: -- [~drankye] Thanks for your suggestions! I have modified the code according to your comments: 1. Done 2. Done 3. The check is used to make sure that when a file is deleted, the parent directory still exists. for example, when a file named "/temp/tests/test0" is deleted, and if parent directory "/temp/tests/" doesn't exist, then it will be created as parent directory 4. We keep the method name not changed, it is used to check whether the path is valid, if "/temp/tests" exists and the size is not zero, then "/temp/tests/test0" is not a valid file name. 6. we don't have strong intentions to do it currently, because most of the logic can not be reused, and many parameters are needed for the utility functions was (Author: shimingfei): [~drankye] Thanks for your suggestions! I have modified the code according to your comments: 1. Done 2. Done 3. The check is used to make sure that when a file is deleted, the parent directory still exists. for example, when a file named "/temp/tests/test0" is deleted, and if parent directory "/temp/tests/" doesn't exist, then it will be created as parent directory 4. We keep the method name not changed, it is used to check whether the path is valid, if "/temp/tests" exists and the size is not zero, then "/temp/tests/test0" is not a valid file name. 6. we don't have strong intentions to do it currently, because most of the logic can not be reused, and many parameters are needed for the utility functions Since the functionality and testing are done, I suggest merging this first, and Aliyun will do the optimization work in following PRs > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HADOOP-12756.003.patch, > HADOOP-12756.004.patch, HADOOP-12756.005.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
