[GitHub] [hadoop-ozone] prashantpogde opened a new pull request #1279: HDDS-3446. Enable TestOzoneManagerRestart and address any failure.

2020-07-30 Thread GitBox
prashantpogde opened a new pull request #1279: URL: https://github.com/apache/hadoop-ozone/pull/1279 ## What changes were proposed in this pull request? Enable and address failure for TestOzoneManagerRestart test cases. ## What is the link to the Apache JIRA https://issu

[jira] [Updated] (HDDS-3446) Enable TestOzoneManagerRestart test cases

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3446: - Labels: pull-request-available (was: ) > Enable TestOzoneManagerRestart test cases >

[jira] [Updated] (HDDS-4053) Volume space quota: add quotaUsageInBytes and update it when create key and file

2020-07-30 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao updated HDDS-4053: Summary: Volume space quota: add quotaUsageInBytes and update it when create key and file (was: Vol

[jira] [Assigned] (HDDS-4053) Volume quota: add quotaUsageInBytes and update it when create key and file

2020-07-30 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-4053: --- Assignee: mingchao zhao > Volume quota: add quotaUsageInBytes and update it when create key a

[jira] [Resolved] (HDDS-3421) Enable TestStorageContainerManager test cases

2020-07-30 Thread Prashant Pogde (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Pogde resolved HDDS-3421. -- Resolution: Fixed > Enable TestStorageContainerManager test cases >

[GitHub] [hadoop-ozone] maobaolong closed pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload

2020-07-30 Thread GitBox
maobaolong closed pull request #1228: URL: https://github.com/apache/hadoop-ozone/pull/1228 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1231: HDDS-3994. Make retry policy can be set by configuration.

2020-07-30 Thread GitBox
maobaolong commented on pull request #1231: URL: https://github.com/apache/hadoop-ozone/pull/1231#issuecomment-666949820 @xiaoyuyao @lokeshj1703 Thanks for the discussion and glad to make consensual for us. Now, I change the default policy to the newer RequestTypeDependentRetryPolicy, PTAL

[jira] [Commented] (HDDS-3421) Enable TestStorageContainerManager test cases

2020-07-30 Thread Prashant Pogde (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168422#comment-17168422 ] Prashant Pogde commented on HDDS-3421: -- Additional failure found is addressed under H

[jira] [Commented] (HDDS-3834) We need a edge(master) docs website updated by nightly build

2020-07-30 Thread maobaolong (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168421#comment-17168421 ] maobaolong commented on HDDS-3834: -- [~elek] I submit a PR to the apache-site project, ht

[jira] [Created] (HDDS-4053) Volume quota: add quotaUsageInBytes and update it when create key and file

2020-07-30 Thread mingchao zhao (Jira)
mingchao zhao created HDDS-4053: --- Summary: Volume quota: add quotaUsageInBytes and update it when create key and file Key: HDDS-4053 URL: https://issues.apache.org/jira/browse/HDDS-4053 Project: Hadoop

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
captainzmc commented on pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#issuecomment-666932982 > Just a question will this change cause any compatibility issues? (As this is new API, only new client code will use it, just seeing if I missed anything) Thanks

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
captainzmc commented on a change in pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#discussion_r463410723 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysRenameRequest.java ## @@ -0,0 +1,279 @@ +/**

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
bharatviswa504 commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666925792 > > As mentioned in the PR description user can use bucket links. > > For more information refer [HDDS-3612](https://issues.apache.org/jira/browse/HDDS-3612). >

[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
timmylicheng commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666924040 > As mentioned in the PR description user can use bucket links. > > For more information refer [HDDS-3612](https://issues.apache.org/jira/browse/HDDS-3612). >

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
bharatviswa504 commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666923085 As mentioned in the PR description user can use bucket links. For more information refer HDDS-3612. This feature is totally not removed, making this hidd

[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
timmylicheng commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666921618 How do we handle legacy volumes and buckets? We used to have volumes mapping to s3 bucket names? So after this change, if a cluster want to support both new buckets

[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #1258: HDDS-4029. Recon unable to add a new container which is in CLOSED state.

2020-07-30 Thread GitBox
github-actions[bot] commented on pull request #1258: URL: https://github.com/apache/hadoop-ozone/pull/1258#issuecomment-666919507 To re-run CI checks, please follow these steps with the source branch checked out: ``` git commit --allow-empty -m 'trigger new CI check' git push ``

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1258: HDDS-4029. Recon unable to add a new container which is in CLOSED state.

2020-07-30 Thread GitBox
avijayanhwx commented on pull request #1258: URL: https://github.com/apache/hadoop-ozone/pull/1258#issuecomment-666919370 /retest This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
bharatviswa504 commented on a change in pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#discussion_r463402693 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysRenameRequest.java ## @@ -0,0 +1,279 @@ +

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1096: HDDS-3833. Use Pipeline choose policy to choose pipeline from exist pipeline list

2020-07-30 Thread GitBox
maobaolong commented on pull request #1096: URL: https://github.com/apache/hadoop-ozone/pull/1096#issuecomment-666908577 @avijayanhwx @elek I've addressed all your comments and add some ut, PTAL. This is an automated message

[GitHub] [hadoop-ozone] xiaoyuyao edited a comment on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
xiaoyuyao edited a comment on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666904747 +1 for this change. Keep exposing this can be problematic after OM create default s3v volume based on the same configuration upon start. If this got changed and th

[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
xiaoyuyao commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666904747 +1 for this change. Keep exposing this after OM create default s3v volume based on the same configuration upon start. If this got changed and then all the buckets in the

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1264: HDDS-4035. Update logs of HadoopDirGenerator.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1264: URL: https://github.com/apache/hadoop-ozone/pull/1264#discussion_r463393431 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/BaseFreonGenerator.java ## @@ -301,6 +301,18 @@ public void printRepo

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1269: HDDS-4042. Update documentation for the GA release

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1269: URL: https://github.com/apache/hadoop-ozone/pull/1269#discussion_r463156542 ## File path: hadoop-hdds/docs/content/concept/OzoneManager.md ## @@ -21,6 +24,8 @@ summary: Ozone Manager is the principal name space service of

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191#discussion_r463351164 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManagerV2Impl.java ## @@ -431,15 +464,21 @@ public

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191#discussion_r463350598 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManagerV2Impl.java ## @@ -136,6 +139,9 @@ public st

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191#discussion_r463350043 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NonHealthyToHealthyNodeHandler.java ## @@ -42,6 +47,11 @@ publi

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191#discussion_r463349967 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NewNodeHandler.java ## @@ -41,6 +46,11 @@ public NewNodeHandler

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191#discussion_r463349579 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMRatisServerImpl.java ## @@ -110,4 +111,18 @@ public void stop(

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1191: HDDS-3837 Add isLeader check in SCMHAManager.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1191: URL: https://github.com/apache/hadoop-ozone/pull/1191#discussion_r463348101 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAManagerImpl.java ## @@ -67,6 +101,44 @@ public SCMRatisServe

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1258: HDDS-4029. Recon unable to add a new container which is in CLOSED state.

2020-07-30 Thread GitBox
avijayanhwx commented on a change in pull request #1258: URL: https://github.com/apache/hadoop-ozone/pull/1258#discussion_r463338176 ## File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconContainerManager.java ## @@ -86,12 +104,39 @@ public v

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1258: HDDS-4029. Recon unable to add a new container which is in CLOSED state.

2020-07-30 Thread GitBox
avijayanhwx commented on a change in pull request #1258: URL: https://github.com/apache/hadoop-ozone/pull/1258#discussion_r463337964 ## File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/AbstractReconContainerManagerTest.java ## @@ -129,4 +132,46 @@

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1258: HDDS-4029. Recon unable to add a new container which is in CLOSED state.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1258: URL: https://github.com/apache/hadoop-ozone/pull/1258#discussion_r463335640 ## File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/TestReconContainerManager.java ## @@ -86,12 +104,39 @@ public voi

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1258: HDDS-4029. Recon unable to add a new container which is in CLOSED state.

2020-07-30 Thread GitBox
xiaoyuyao commented on a change in pull request #1258: URL: https://github.com/apache/hadoop-ozone/pull/1258#discussion_r463334255 ## File path: hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/scm/AbstractReconContainerManagerTest.java ## @@ -129,4 +132,46 @@ pr

[jira] [Commented] (HDDS-4047) OzoneManager met NPE exception while getServiceList

2020-07-30 Thread maobaolong (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168307#comment-17168307 ] maobaolong commented on HDDS-4047: -- [~xyao] Thanks for your review. > OzoneManager met N

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1277: HDDS-4047. OzoneManager met NPE exception while getServiceList

2020-07-30 Thread GitBox
maobaolong commented on pull request #1277: URL: https://github.com/apache/hadoop-ozone/pull/1277#issuecomment-666776047 @xiaoyuyao Thanks for your review and merge this for me. This is an automated message from the Apache Gi

[jira] [Resolved] (HDDS-4047) OzoneManager met NPE exception while getServiceList

2020-07-30 Thread Xiaoyu Yao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDDS-4047. -- Fix Version/s: 0.6.0 Resolution: Fixed Thanks [~maobaolong] for the contribution. Patch has been

[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #1277: HDDS-4047. OzoneManager met NPE exception while getServiceList

2020-07-30 Thread GitBox
xiaoyuyao merged pull request #1277: URL: https://github.com/apache/hadoop-ozone/pull/1277 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Updated] (HDDS-4050) Remove non-inclusive terminology from Ozone

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDDS-4050: - Description: [http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/2

[jira] [Updated] (HDDS-4052) Remove master/slave terminology from Ozone

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDDS-4052: - Summary: Remove master/slave terminology from Ozone (was: CLONE - Remove

[jira] [Updated] (HDDS-4051) Remove whitelist/blacklist terminology from Ozone

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian updated HDDS-4051: - Summary: Remove whitelist/blacklist terminology from Ozone (was: CLONE -

[jira] [Assigned] (HDDS-4052) Remove master/slave terminology from Ozone

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian reassigned HDDS-4052: Assignee: (was: Vivek Ratnavel Subramanian) > Remove master/sla

[jira] [Assigned] (HDDS-4050) Remove non-inclusive terminology from Ozone

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian reassigned HDDS-4050: Assignee: Vivek Ratnavel Subramanian > Remove non-inclusive termino

[jira] [Assigned] (HDDS-4052) CLONE - Remove master/slave terminology from Hadoop

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian reassigned HDDS-4052: Key: HDDS-4052 (was: HADOOP-17174) Workflow: patch-availab

[jira] [Assigned] (HDDS-4051) CLONE - Remove whitelist/blacklist terminology from Hadoop

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian reassigned HDDS-4051: Key: HDDS-4051 (was: HADOOP-17173) Workflow: patch-availab

[jira] [Moved] (HDDS-4050) Remove non-inclusive terminology from Ozone

2020-07-30 Thread Vivek Ratnavel Subramanian (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Ratnavel Subramanian moved HADOOP-17172 to HDDS-4050: --- Key: HDDS-4050 (was: HADOOP-17172) W

[GitHub] [hadoop-ozone] elek commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

2020-07-30 Thread GitBox
elek commented on pull request #1255: URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-70426 > The only reason we 'tag' the name of the DB with the timestamp is to help with troubleshooting Recon - OM sync problems... Oh, got it. Thanks to explain it. Sounds rea

[jira] [Assigned] (HDDS-3446) Enable TestOzoneManagerRestart test cases

2020-07-30 Thread Prashant Pogde (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Pogde reassigned HDDS-3446: Assignee: Prashant Pogde > Enable TestOzoneManagerRestart test cases >

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

2020-07-30 Thread GitBox
avijayanhwx commented on pull request #1255: URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-666542216 > I am still not sure about this one (but as I wrote I am not familiar). In case of a new snapshot: why don't we drop the previous one and start to use a new one? Why d

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

2020-07-30 Thread GitBox
avijayanhwx commented on a change in pull request #1255: URL: https://github.com/apache/hadoop-ozone/pull/1255#discussion_r463151520 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBDefinitionFactory.java ## @@ -40,10 +46,24 @@ private DBDefinitio

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

2020-07-30 Thread GitBox
avijayanhwx commented on a change in pull request #1255: URL: https://github.com/apache/hadoop-ozone/pull/1255#discussion_r463151665 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBScanner.java ## @@ -44,28 +44,46 @@ description = "T

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1096: HDDS-3833. Use Pipeline choose policy to choose pipeline from exist pipeline list

2020-07-30 Thread GitBox
avijayanhwx commented on pull request #1096: URL: https://github.com/apache/hadoop-ozone/pull/1096#issuecomment-666539295 @maobaolong There are still some unaddressed comments here. - Using Java based config API - Adding unit test(s). PTAL -

[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1212: HDDS-3979. Make bufferSize configurable for stream copy

2020-07-30 Thread GitBox
maobaolong commented on a change in pull request #1212: URL: https://github.com/apache/hadoop-ozone/pull/1212#discussion_r463120834 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java ## @@ -465,6 +465,12 @@ public static final String

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1212: HDDS-3979. Make bufferSize configurable for stream copy

2020-07-30 Thread GitBox
maobaolong commented on pull request #1212: URL: https://github.com/apache/hadoop-ozone/pull/1212#issuecomment-666507055 @elek Thanks you for your review and suggestion, i've addressed your comments, PTAL. This is an automa

[jira] [Created] (HDDS-4049) Use annotation based configuration to replace the s3g config

2020-07-30 Thread maobaolong (Jira)
maobaolong created HDDS-4049: Summary: Use annotation based configuration to replace the s3g config Key: HDDS-4049 URL: https://issues.apache.org/jira/browse/HDDS-4049 Project: Hadoop Distributed Data Sto

[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1212: HDDS-3979. Make bufferSize configurable for stream copy

2020-07-30 Thread GitBox
maobaolong commented on a change in pull request #1212: URL: https://github.com/apache/hadoop-ozone/pull/1212#discussion_r463113923 ## File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java ## @@ -114,6 +120,13 @@ public ObjectE

[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1214: HDDS-3981. Add more debug level log to XceiverClientGrpc for debug purpose

2020-07-30 Thread GitBox
maobaolong commented on a change in pull request #1214: URL: https://github.com/apache/hadoop-ozone/pull/1214#discussion_r462705761 ## File path: hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java ## @@ -354,7 +363,7 @@ private XceiverClientRepl

[GitHub] [hadoop-ozone] maobaolong closed pull request #1277: HDDS-4047. OzoneManager met NPE exception while getServiceList

2020-07-30 Thread GitBox
maobaolong closed pull request #1277: URL: https://github.com/apache/hadoop-ozone/pull/1277 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1278: HDDS-4048. Show more information while SCM version info mismatch

2020-07-30 Thread GitBox
maobaolong opened a new pull request #1278: URL: https://github.com/apache/hadoop-ozone/pull/1278 ## What changes were proposed in this pull request? Now, if I accident to connect unexpected SCM since i specify the wrong ozone-site.xml file, the OM cannot startup and through exceptio

[jira] [Updated] (HDDS-4048) Show more information while SCM version info mismatch

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4048: - Labels: pull-request-available (was: ) > Show more information while SCM version info mismatch >

[jira] [Created] (HDDS-4048) Show more information while SCM version info mismatch

2020-07-30 Thread maobaolong (Jira)
maobaolong created HDDS-4048: Summary: Show more information while SCM version info mismatch Key: HDDS-4048 URL: https://issues.apache.org/jira/browse/HDDS-4048 Project: Hadoop Distributed Data Store

[jira] [Updated] (HDDS-4047) OzoneManager met NPE exception while getServiceList

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4047: - Labels: pull-request-available (was: ) > OzoneManager met NPE exception while getServiceList > --

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1277: HDDS-4047. OzoneManager met NPE exception while getServiceList

2020-07-30 Thread GitBox
maobaolong commented on pull request #1277: URL: https://github.com/apache/hadoop-ozone/pull/1277#issuecomment-666455316 This has been test locally. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1277: HDDS-4047. OzoneManager met NPE exception while getServiceList

2020-07-30 Thread GitBox
maobaolong opened a new pull request #1277: URL: https://github.com/apache/hadoop-ozone/pull/1277 ## What changes were proposed in this pull request? OzoneManager met NPE exception while getServiceList after just startup. ## What is the link to the Apache JIRA HDDS-4047

[jira] [Created] (HDDS-4047) OzoneManager met NPE exception while getServiceList

2020-07-30 Thread maobaolong (Jira)
maobaolong created HDDS-4047: Summary: OzoneManager met NPE exception while getServiceList Key: HDDS-4047 URL: https://issues.apache.org/jira/browse/HDDS-4047 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
captainzmc commented on pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#issuecomment-666395020 > Thanks @captainzmc for changing the protocol. I think it is much improved now. I only have couple of minor comments left. Thanks for @adoroszlai 's feedback. Up

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
adoroszlai commented on a change in pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#discussion_r463010326 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysRenameRequest.java ## @@ -0,0 +1,279 @@ +/**

[GitHub] [hadoop-ozone] elek commented on pull request #1083: HDDS-3814. Drop a column family through debug cli tool

2020-07-30 Thread GitBox
elek commented on pull request #1083: URL: https://github.com/apache/hadoop-ozone/pull/1083#issuecomment-666371407 So what is the conclusion here. Majority seems to have concerns about providing generic R/W RocksDB tools. I created a related patch: #1276. It makes it possible t

[GitHub] [hadoop-ozone] elek opened a new pull request #1276: HDDS-4046. Extensible subcommands for CLI applications

2020-07-30 Thread GitBox
elek opened a new pull request #1276: URL: https://github.com/apache/hadoop-ozone/pull/1276 ## What changes were proposed in this pull request? HDDS-3814 proposed a new subcommand which deletes the column families from the rocksdb tables. But during the discussion ther

[jira] [Updated] (HDDS-4046) Extensible subcommands for CLI applications

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4046: - Labels: pull-request-available (was: ) > Extensible subcommands for CLI applications > --

[jira] [Created] (HDDS-4046) Extensible subcommands for CLI applications

2020-07-30 Thread Marton Elek (Jira)
Marton Elek created HDDS-4046: - Summary: Extensible subcommands for CLI applications Key: HDDS-4046 URL: https://issues.apache.org/jira/browse/HDDS-4046 Project: Hadoop Distributed Data Store Iss

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
captainzmc commented on a change in pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#discussion_r462993999 ## File path: hadoop-ozone/interface-client/src/main/resources/proto.lock ## @@ -83,6 +83,10 @@ "name": "DeleteKeys",

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1150: HDDS-3903. OzoneRpcClient support batch rename keys.

2020-07-30 Thread GitBox
captainzmc commented on a change in pull request #1150: URL: https://github.com/apache/hadoop-ozone/pull/1150#discussion_r462993785 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -750,6 +751,25 @@ public void renameKey(S

[jira] [Updated] (HDDS-3990) Test Kubernetes examples with acceptance tests

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3990: - Labels: pull-request-available (was: ) > Test Kubernetes examples with acceptance tests > ---

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1223: HDDS-3990. Test Kubernetes examples with acceptance tests

2020-07-30 Thread GitBox
adoroszlai commented on pull request #1223: URL: https://github.com/apache/hadoop-ozone/pull/1223#issuecomment-666332434 Thanks @elek for improving it further. This is an automated message from the Apache Git Service. To resp

[jira] [Resolved] (HDDS-3990) Test Kubernetes examples with acceptance tests

2020-07-30 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai resolved HDDS-3990. Fix Version/s: 0.7.0 Resolution: Implemented > Test Kubernetes examples with acceptan

[jira] [Updated] (HDDS-3990) Test Kubernetes examples with acceptance tests

2020-07-30 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-3990: --- Labels: (was: pull-request-available) > Test Kubernetes examples with acceptance tests > ---

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1223: HDDS-3990. Test Kubernetes examples with acceptance tests

2020-07-30 Thread GitBox
adoroszlai merged pull request #1223: URL: https://github.com/apache/hadoop-ozone/pull/1223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

2020-07-30 Thread GitBox
elek commented on a change in pull request #1255: URL: https://github.com/apache/hadoop-ozone/pull/1255#discussion_r462949586 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBDefinitionFactory.java ## @@ -40,10 +46,24 @@ private DBDefinitionFactor

[GitHub] [hadoop-ozone] elek commented on pull request #1255: HDDS-4021. Organize Recon DBs into a 'DBDefinition'.

2020-07-30 Thread GitBox
elek commented on pull request #1255: URL: https://github.com/apache/hadoop-ozone/pull/1255#issuecomment-666323762 > The container key DB is a by product of processing the OM snapshot DB. It hosts the reverse mapping of the OM data, the Container->Key mapping in a specific format Th

[jira] [Updated] (HDDS-4045) Add more ignore rules to the RAT ignore list

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4045: - Labels: pull-request-available (was: ) > Add more ignore rules to the RAT ignore list > -

[jira] [Updated] (HDDS-4045) Add more ignore rules to the RAT ignore list

2020-07-30 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4045: --- Labels: (was: pull-request-available) > Add more ignore rules to the RAT ignore list > -

[GitHub] [hadoop-ozone] elek commented on pull request #1270: HDDS-4044. Deprecate ozone.s3g.volume.name.

2020-07-30 Thread GitBox
elek commented on pull request #1270: URL: https://github.com/apache/hadoop-ozone/pull/1270#issuecomment-666318907 > More flexibility usually comes with more complexity. :) In general, I can agree with this statement, but in this specific case: the additional complexity of one confi

[jira] [Resolved] (HDDS-4045) Add more ignore rules to the RAT ignore list

2020-07-30 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai resolved HDDS-4045. Fix Version/s: 0.7.0 Resolution: Fixed > Add more ignore rules to the RAT ignore list

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1273: HDDS-4045. Add more ignore rules to the RAT ignore list

2020-07-30 Thread GitBox
adoroszlai merged pull request #1273: URL: https://github.com/apache/hadoop-ozone/pull/1273 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] elek commented on pull request #1206: HDDS-3972. Add option to limit number of items displaying through ldb tool.

2020-07-30 Thread GitBox
elek commented on pull request #1206: URL: https://github.com/apache/hadoop-ozone/pull/1206#issuecomment-666305595 Thanks the update @sadanand48. Yes it looks better (and significant better) for me. I realized two other NITs in the test. 1. `@Test(expected = IllegalArgumentExc

[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

2020-07-30 Thread GitBox
aryangupta1998 commented on a change in pull request #1266: URL: https://github.com/apache/hadoop-ozone/pull/1266#discussion_r462904139 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopNestedDirGenerator.java ## @@ -0,0 +1,208 @

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1223: HDDS-3990. Test Kubernetes examples with acceptance tests

2020-07-30 Thread GitBox
elek commented on a change in pull request #1223: URL: https://github.com/apache/hadoop-ozone/pull/1223#discussion_r462897010 ## File path: .github/workflows/post-commit.yml ## @@ -238,3 +238,61 @@ jobs: name: coverage path: target/coverage con

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1223: HDDS-3990. Test Kubernetes examples with acceptance tests

2020-07-30 Thread GitBox
elek commented on a change in pull request #1223: URL: https://github.com/apache/hadoop-ozone/pull/1223#discussion_r462886907 ## File path: hadoop-ozone/dist/src/main/k8s/examples/getting-started/test.sh ## @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Licensed to the Apache Softwar

[GitHub] [hadoop-ozone] elek commented on pull request #1273: HDDS-4045. Add more ignore rules to the RAT ignore list

2020-07-30 Thread GitBox
elek commented on pull request #1273: URL: https://github.com/apache/hadoop-ozone/pull/1273#issuecomment-666269550 BTW, it would be useful to disable some of the tests at some specific case. For example, I am not sure if I need full retest for this patch/commit. --

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1273: HDDS-4045. Add more ignore rules to the RAT ignore list

2020-07-30 Thread GitBox
elek commented on a change in pull request #1273: URL: https://github.com/apache/hadoop-ozone/pull/1273#discussion_r462884361 ## File path: hadoop-hdds/pom.xml ## @@ -252,7 +252,15 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd";> **/hs_err*.

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-07-30 Thread GitBox
adoroszlai commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r462883821 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -821,6 +821,13 @@ Number of containers per owner per disk in a pi

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1273: HDDS-4045. Add more ignore rules to the RAT ignore list

2020-07-30 Thread GitBox
adoroszlai commented on a change in pull request #1273: URL: https://github.com/apache/hadoop-ozone/pull/1273#discussion_r462883261 ## File path: hadoop-hdds/pom.xml ## @@ -252,7 +252,15 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd";> **/hs

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1274: HDDS-3810. Add the logic to distribute open containers among the pipelines of a datanode.

2020-07-30 Thread GitBox
adoroszlai commented on a change in pull request #1274: URL: https://github.com/apache/hadoop-ozone/pull/1274#discussion_r462876760 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java ## @@ -307,6 +307,10 @@ OZONE_SCM_KEY_VALUE_C

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1266: HDDS-4034. Add Unit Test for HadoopNestedDirGenerator.

2020-07-30 Thread GitBox
bshashikant commented on a change in pull request #1266: URL: https://github.com/apache/hadoop-ozone/pull/1266#discussion_r462877851 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestHadoopNestedDirGenerator.java ## @@ -0,0 +1,208 @@ +

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1192: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-07-30 Thread GitBox
bshashikant commented on a change in pull request #1192: URL: https://github.com/apache/hadoop-ozone/pull/1192#discussion_r462864723 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -1016,6 +1016,7 @@ public OzoneFileStatu

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1192: HDDS-3947: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-07-30 Thread GitBox
bshashikant commented on a change in pull request #1192: URL: https://github.com/apache/hadoop-ozone/pull/1192#discussion_r462864723 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java ## @@ -1016,6 +1016,7 @@ public OzoneFileStatu

[GitHub] [hadoop-ozone] bshashikant commented on pull request #1154: HDDS-3867. Extend the chunkinfo tool to display information from all nodes in the pipeline.

2020-07-30 Thread GitBox
bshashikant commented on pull request #1154: URL: https://github.com/apache/hadoop-ozone/pull/1154#issuecomment-666245556 > Thanks to create this patch @sadanand48 > > I have one design concern related to this change. I am not sure what is the good answer, but trying to share my view

[jira] [Updated] (HDDS-4009) Recon Overview page: The volume, bucket and key counts are not accurate

2020-07-30 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-4009: - Labels: pull-request-available (was: ) > Recon Overview page: The volume, bucket and key counts a

[GitHub] [hadoop-ozone] vivekratnavel opened a new pull request #1275: HDDS-4009. Recon Overview page: The volume, bucket and key counts are not accurate

2020-07-30 Thread GitBox
vivekratnavel opened a new pull request #1275: URL: https://github.com/apache/hadoop-ozone/pull/1275 ## What changes were proposed in this pull request? Instead of relying on Rocks DB's "rocksdb.estimate-num-keys" to get the counts of volume, bucket, and key, a new task is added to p

  1   2   >