[GitHub] [hadoop-ozone] captainzmc commented on pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-27 Thread GitBox
captainzmc commented on pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#issuecomment-699612552 Thanks @ChenSammi for the review. Review issues has been fixed. This is an automated message from the Ap

[GitHub] [hadoop-ozone] captainzmc removed a comment on pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-27 Thread GitBox
captainzmc removed a comment on pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#issuecomment-698235132 Thanks for @ChenSammi ‘s review. The review issues have been fixed. Could you help take another look? -

[GitHub] [hadoop-ozone] timmylicheng closed pull request #1340: HDDS-3188 Add failover proxy for SCM block location.

2020-09-27 Thread GitBox
timmylicheng closed pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] timmylicheng closed pull request #1340: HDDS-3188 Add failover proxy for SCM block location.

2020-09-27 Thread GitBox
timmylicheng closed pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-27 Thread GitBox
timmylicheng commented on pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#issuecomment-699599648 Thanks @amaliujia for the contribution. The patch looks good overall. Just a few comments inline. Thi

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-27 Thread GitBox
timmylicheng commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495539471 ## File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/MockSCMHAManager.java ## @@ -107,6 +108,14 @@ public void shut

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-27 Thread GitBox
timmylicheng commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495538752 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestScmAdminHA.java ## @@ -0,0 +1,79 @@ +/* + * License

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-27 Thread GitBox
captainzmc commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495538695 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java ## @@ -277,6 +277,16 @@ p

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-27 Thread GitBox
timmylicheng commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495538644 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/ScmAdmin.java ## @@ -0,0 +1,60 @@ +/* + * Licensed to the Apac

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1346: HDDS-4115. CLI command to show current SCM leader and follower status.

2020-09-27 Thread GitBox
timmylicheng commented on a change in pull request #1346: URL: https://github.com/apache/hadoop-ozone/pull/1346#discussion_r495538535 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/scm/GetScmRatisStatusSubcommand.java ## @@ -0,0 +1,46 @@ +/* + * L

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1428: HDDS-4192: enable SCM Raft Group based on config ozone.scm.names

2020-09-27 Thread GitBox
timmylicheng commented on a change in pull request #1428: URL: https://github.com/apache/hadoop-ozone/pull/1428#discussion_r495538313 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMRatisServerImpl.java ## @@ -59,18 +68,62 @@ SCMRatisServ

[GitHub] [hadoop-ozone] timmylicheng commented on pull request #1340: HDDS-3188 Add failover proxy for SCM block location.

2020-09-26 Thread GitBox
timmylicheng commented on pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340#issuecomment-699594942 > Thanks @timmylicheng for working on this. The patch LGTM overall, a few comments added inline. @xiaoyuyao Thanks for the reviews! I've updated the PR. Please t

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1340: HDDS-3188 Add failover proxy for SCM block location.

2020-09-26 Thread GitBox
timmylicheng commented on a change in pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340#discussion_r495537521 ## File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMProxyInfo.java ## @@ -0,0 +1,65 @@ +/** + * Licensed to t

[GitHub] [hadoop-ozone] captainzmc removed a comment on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-26 Thread GitBox
captainzmc removed a comment on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-697133030 Thanks @ChenSammi for the review. Review issues has been fixed. This is an automated message fro

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-26 Thread GitBox
captainzmc commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-699594512 Thanks @ChenSammi for the review. Review issues has been fixed. This is an automated message from the Ap

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1340: HDDS-3188 Add failover proxy for SCM block location.

2020-09-26 Thread GitBox
timmylicheng commented on a change in pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340#discussion_r495537306 ## File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMClientConfig.java ## @@ -0,0 +1,107 @@ +/* + * Licensed t

[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1340: HDDS-3188 Add failover proxy for SCM block location.

2020-09-26 Thread GitBox
timmylicheng commented on a change in pull request #1340: URL: https://github.com/apache/hadoop-ozone/pull/1340#discussion_r495537270 ## File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/proxy/SCMClientConfig.java ## @@ -0,0 +1,107 @@ +/* + * Licensed t

[GitHub] [hadoop-ozone] captainzmc closed pull request #1446: HDDS-4274. Change the log level of the SCM Delete block to improve performance.

2020-09-26 Thread GitBox
captainzmc closed pull request #1446: URL: https://github.com/apache/hadoop-ozone/pull/1446 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-26 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521898 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java ## @@ -239,8 +256,7 @@ p

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-26 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521569 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java ## @@ -533,6 +533,26 @@ protected

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-26 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521253 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java ## @@ -707,9 +707,71

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-26 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495521089 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMAllocateBlockRequest.java ## @@ -227,6 +232,18 @@

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-26 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r495520762 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java ## @@ -277,6 +277,16 @@ pu

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-26 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r495519238 ## File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto ## @@ -314,6 +314,8 @@ enum Status { PARTIAL_RENAME = 65;

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1446: HDDS-4274. Change the log level of the SCM Delete block to improve performance.

2020-09-26 Thread GitBox
captainzmc commented on a change in pull request #1446: URL: https://github.com/apache/hadoop-ozone/pull/1446#discussion_r495517233 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMBlockProtocolServer.java ## @@ -218,7 +218,7 @@ public vo

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1446: HDDS-4274. Change the log level of the SCM Delete block to improve performance.

2020-09-26 Thread GitBox
amaliujia commented on a change in pull request #1446: URL: https://github.com/apache/hadoop-ozone/pull/1446#discussion_r495508443 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMBlockProtocolServer.java ## @@ -218,7 +218,7 @@ public voi

[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1449: HDDS-4273

2020-09-26 Thread GitBox
amaliujia opened a new pull request #1449: URL: https://github.com/apache/hadoop-ozone/pull/1449 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## What is the link to the Apache JIRA (Please create an issue in ASF JIRA b

[GitHub] [hadoop-ozone] linyiqun opened a new pull request #1448: HDDS-4280. Document notable configurations for Recon.

2020-09-26 Thread GitBox
linyiqun opened a new pull request #1448: URL: https://github.com/apache/hadoop-ozone/pull/1448 ## What changes were proposed in this pull request? Document notable configurations for Recon ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1447: HDDS-4264. Uniform naming conventions of Ozone Shell Options.

2020-09-25 Thread GitBox
amaliujia commented on pull request #1447: URL: https://github.com/apache/hadoop-ozone/pull/1447#issuecomment-699255649 +1 on unifying the name convention. Though not sure which name convention is better :) Would also want to hear suggestions from @adoroszlai -

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-25 Thread GitBox
ChenSammi commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-698899914 > Sorry for the slow reply on this. I have been caught up on some other things. > > > After a second thought, deleting the container record in SCM DB immediately w

[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1447: HDDS-4264. Uniform naming conventions of Ozone Shell Options.

2020-09-25 Thread GitBox
captainzmc opened a new pull request #1447: URL: https://github.com/apache/hadoop-ozone/pull/1447 ## What changes were proposed in this pull request? Current Shell command of Ozone, some use hump connection, some use '-' connection. We need to unify the naming conventions. See the

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1447: HDDS-4264. Uniform naming conventions of Ozone Shell Options.

2020-09-25 Thread GitBox
captainzmc commented on pull request #1447: URL: https://github.com/apache/hadoop-ozone/pull/1447#issuecomment-698921171 Hi @adoroszlai,Could you help review this PR? This is an automated message from the Apache Git Service.

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-25 Thread GitBox
captainzmc commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494297018 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/bucket/OMBucketDeleteResponse.java ## @@ -64,6 +76,12

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-25 Thread GitBox
amaliujia commented on pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#issuecomment-698104826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1383: HDDS-4194. Create a script to check AWS S3 compatibility

2020-09-25 Thread GitBox
adoroszlai merged pull request #1383: URL: https://github.com/apache/hadoop-ozone/pull/1383 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-25 Thread GitBox
amaliujia commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494575654 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java ## @@ -201,6 +201,8 @@

[GitHub] [hadoop-ozone] cxorm commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-25 Thread GitBox
cxorm commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-698096286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-25 Thread GitBox
adoroszlai merged pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] sodonnel commented on a change in pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-25 Thread GitBox
sodonnel commented on a change in pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#discussion_r494180772 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java ## @@ -96,13 +101,24 @@

[GitHub] [hadoop-ozone] bshashikant merged pull request #1442: HDDS-3297. Enable TestOzoneClientKeyGenerator.

2020-09-25 Thread GitBox
bshashikant merged pull request #1442: URL: https://github.com/apache/hadoop-ozone/pull/1442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-25 Thread GitBox
captainzmc commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-698132943 Hi all, status updates: rebase PR and resolve conflicts. This PR can be reviewed again. This is an aut

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-25 Thread GitBox
adoroszlai merged pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-25 Thread GitBox
elek commented on a change in pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443#discussion_r494239916 ## File path: dev-support/byteman/watchforcommit_all.btm ## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# co

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-25 Thread GitBox
captainzmc commented on pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#issuecomment-698235132 Thanks for @ChenSammi ‘s review. The review issues have been fixed. Could you help take another look?

[GitHub] [hadoop-ozone] cxorm commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-25 Thread GitBox
cxorm commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-698094498 Thanks @captainzmc for the work. Would you please rebase this PR ? I would take a look on it. This

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-25 Thread GitBox
adoroszlai commented on pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443#issuecomment-698385899 Thanks @elek for updating the patch. This is an automated message from the Apache Git Service. To respon

[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-25 Thread GitBox
cxorm commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494095636 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmVolumeArgs.java ## @@ -68,15 +69,16 @@ "builder."}) pri

[GitHub] [hadoop-ozone] sodonnel commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-25 Thread GitBox
sodonnel commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-698259315 Sorry for the slow reply on this. I have been caught up on some other things. > After a second thought, deleting the container record in SCM DB immediately while kee

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1298: HDDS-3869. Use different column families for datanode block and metadata

2020-09-25 Thread GitBox
hanishakoneru commented on pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298#issuecomment-698555281 @errose28, the unit test failures seem to be related to the patch. Can you please check. This is an

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-25 Thread GitBox
ChenSammi commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-698899914 > Sorry for the slow reply on this. I have been caught up on some other things. > > > After a second thought, deleting the container record in SCM DB immediately w

[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1446: HDDS-4274. Change the log level of the SCM Delete block to improve performance.

2020-09-25 Thread GitBox
captainzmc opened a new pull request #1446: URL: https://github.com/apache/hadoop-ozone/pull/1446 ## What changes were proposed in this pull request? There are scenarios in which the DELETE operation is very frequent. Now that info level logs in SCM are affecting performance, we shou

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-24 Thread GitBox
amaliujia commented on pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#issuecomment-698558307 Thanks @cxorm and @captainzmc comments are addressed. Also created https://issues.apache.org/jira/browse/HDDS-4273 to track the work that make `usedNamespa

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-24 Thread GitBox
amaliujia commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494576276 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmVolumeArgs.java ## @@ -68,15 +69,16 @@ "builder."})

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-24 Thread GitBox
amaliujia commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494575654 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java ## @@ -201,6 +201,8 @@

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-24 Thread GitBox
amaliujia commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494575930 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java ## @@ -270,6 +270,7 @@ private OzoneConsts() { public s

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1298: HDDS-3869. Use different column families for datanode block and metadata

2020-09-24 Thread GitBox
hanishakoneru commented on pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298#issuecomment-698555281 @errose28, the unit test failures seem to be related to the patch. Can you please check. This is an

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1383: HDDS-4194. Create a script to check AWS S3 compatibility

2020-09-24 Thread GitBox
adoroszlai merged pull request #1383: URL: https://github.com/apache/hadoop-ozone/pull/1383 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-24 Thread GitBox
adoroszlai commented on pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443#issuecomment-698385899 Thanks @elek for updating the patch. This is an automated message from the Apache Git Service. To respon

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-24 Thread GitBox
adoroszlai merged pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-24 Thread GitBox
captainzmc commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494297018 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/bucket/OMBucketDeleteResponse.java ## @@ -64,6 +76,12

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-24 Thread GitBox
elek commented on a change in pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443#discussion_r494239916 ## File path: dev-support/byteman/watchforcommit_all.btm ## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# co

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-24 Thread GitBox
adoroszlai merged pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [hadoop-ozone] bshashikant merged pull request #1442: HDDS-3297. Enable TestOzoneClientKeyGenerator.

2020-09-24 Thread GitBox
bshashikant merged pull request #1442: URL: https://github.com/apache/hadoop-ozone/pull/1442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] sodonnel commented on pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-24 Thread GitBox
sodonnel commented on pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#issuecomment-698259315 Sorry for the slow reply on this. I have been caught up on some other things. > After a second thought, deleting the container record in SCM DB immediately while kee

[GitHub] [hadoop-ozone] cxorm commented on a change in pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-24 Thread GitBox
cxorm commented on a change in pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#discussion_r494095636 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmVolumeArgs.java ## @@ -68,15 +69,16 @@ "builder."}) pri

[GitHub] [hadoop-ozone] sodonnel commented on a change in pull request #1338: HDDS-4023. Delete closed container after all blocks have been deleted.

2020-09-24 Thread GitBox
sodonnel commented on a change in pull request #1338: URL: https://github.com/apache/hadoop-ozone/pull/1338#discussion_r494180772 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/AbstractContainerReportHandler.java ## @@ -96,13 +101,24 @@

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-24 Thread GitBox
captainzmc commented on pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#issuecomment-698235132 Thanks for @ChenSammi ‘s review. The review issues have been fixed. Could you help take another look?

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-23 Thread GitBox
captainzmc commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-698132943 Hi all, status updates: rebase PR and resolve conflicts. This PR can be reviewed again. This is an aut

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-23 Thread GitBox
amaliujia commented on pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445#issuecomment-698104826 R: @cxorm @ChenSammi @captainzmc Can you take a look please? This is an automated message from th

[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1445: HDDS-4272. Volume namespace: add usedNamespace and update it when create and delete bucket

2020-09-23 Thread GitBox
amaliujia opened a new pull request #1445: URL: https://github.com/apache/hadoop-ozone/pull/1445 ## What changes were proposed in this pull request? Volume namespace: add usedNamespace and update it when create and delete bucket ## What is the link to the Apache JIRA ht

[GitHub] [hadoop-ozone] cxorm commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
cxorm commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-698097298 The work looks great to me. A little question is about the title of the issue, I think the title is too general, could we change it to address the fix of this PR ?

[GitHub] [hadoop-ozone] cxorm commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
cxorm commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-698096286 >In a larger perspective, would it make sense to enhance checkstyle rules to warn for this kind of spacing? Yeah, thanks for the feedback, we could discuss about the id

[GitHub] [hadoop-ozone] cxorm commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-23 Thread GitBox
cxorm commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-698094498 Thanks @captainzmc for the work. Would you please rebase this PR ? I would take a look on it. This

[GitHub] [hadoop-ozone] amaliujia commented on a change in pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia commented on a change in pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444#discussion_r493976795 ## File path: hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/helpers/OzoneAclStorage.java ## @@ -0,0 +1,63 @@ +/** + * Li

[GitHub] [hadoop-ozone] amaliujia edited a comment on pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia edited a comment on pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444#issuecomment-697997438 R @elek Only migrating `PrefixInfo` because I want to see whether my current approach makes sense (e.g. move some classes to `interface-storage` meanwhile

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia commented on pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444#issuecomment-697997438 R @elek This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hadoop-ozone] amaliujia opened a new pull request #1444: HDDS-4242. Copy PrefixInfo proto to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia opened a new pull request #1444: URL: https://github.com/apache/hadoop-ozone/pull/1444 ## What changes were proposed in this pull request? As a following effort, try to move `PrefixInfo` and other necessary classes to `interface-stroage` ## What is the link to the A

[GitHub] [hadoop-ozone] hanishakoneru commented on pull request #1298: HDDS-3869. Use different column families for datanode block and metadata

2020-09-23 Thread GitBox
hanishakoneru commented on pull request #1298: URL: https://github.com/apache/hadoop-ozone/pull/1298#issuecomment-697956338 @errose28, great work on this Jira. +1 for merge pending CI. This is an automated message from the

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread GitBox
adoroszlai commented on a change in pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443#discussion_r493694568 ## File path: dev-support/byteman/ratis-flush.btm ## @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# con

[GitHub] [hadoop-ozone] amaliujia commented on pull request #1424: HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
amaliujia commented on pull request #1424: URL: https://github.com/apache/hadoop-ozone/pull/1424#issuecomment-697702757 @elek thank you for your review! This is an automated message from the Apache Git Service. To respond to

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-23 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r493728718 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java ## @@ -246,4 +246,9 @@ private OMConfigKeys() {

[GitHub] [hadoop-ozone] elek commented on pull request #1319: HDDS-4107: replace scmID with clusterID for container and volume at Datanode side

2020-09-23 Thread GitBox
elek commented on pull request #1319: URL: https://github.com/apache/hadoop-ozone/pull/1319#issuecomment-697593857 (Sorry, late answer) Why do we need to updated container files? I see no problems to keep (the valid) clusterId in the container metadata files. Or do you need t

[GitHub] [hadoop-ozone] elek commented on pull request #1083: HDDS-3814. Drop a column family through debug cli tool

2020-09-23 Thread GitBox
elek commented on pull request #1083: URL: https://github.com/apache/hadoop-ozone/pull/1083#issuecomment-697590296 @bharatviswa504 @avijayanhwx Are you fine with this approach? Command won't be visible unless you put a magic file to the right place (META-INF/services). It's a hidden

[GitHub] [hadoop-ozone] elek merged pull request #1417: HDDS-4324. Add important comment to ListVolumes logic

2020-09-23 Thread GitBox
elek merged pull request #1417: URL: https://github.com/apache/hadoop-ozone/pull/1417 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hadoop-ozone] elek merged pull request #1424: HDDS-4236. Move "Om*Codec.java" to new project hadoop-ozone/interface-storage

2020-09-23 Thread GitBox
elek merged pull request #1424: URL: https://github.com/apache/hadoop-ozone/pull/1424 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-23 Thread GitBox
ChenSammi commented on pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-697328180 LGTM, +1. This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [hadoop-ozone] ChenSammi merged pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-23 Thread GitBox
ChenSammi merged pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [hadoop-ozone] adoroszlai edited a comment on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-23 Thread GitBox
adoroszlai edited a comment on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-697321399 > > I think we should avoid logging chunk content. ContainerCommandRequestMessage implements related logic to clear data. > > I checked but it's not somethi

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-23 Thread GitBox
adoroszlai commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-697321399 > > I think we should avoid logging chunk content. ContainerCommandRequestMessage implements related logic to clear data. > > I checked but it's not something whic

[GitHub] [hadoop-ozone] elek opened a new pull request #1443: HDDS-4270. Add more reusable byteman scripts to debug ofs/o3fs performance

2020-09-23 Thread GitBox
elek opened a new pull request #1443: URL: https://github.com/apache/hadoop-ozone/pull/1443 I am using https://byteman.jboss.org to debug the performance of spark + teragen with different scripts. Some byteman scripts are already shared by HDDS-4095 or HDDS-342 but it seems to be a goo

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1383: HDDS-4194. Create a script to check AWS S3 compatibility

2020-09-23 Thread GitBox
elek commented on a change in pull request #1383: URL: https://github.com/apache/hadoop-ozone/pull/1383#discussion_r493498810 ## File path: hadoop-ozone/dist/src/main/smoketest/s3/s3_compatbility_check.sh ## @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Licensed to the Apache Softwa

[GitHub] [hadoop-ozone] elek commented on a change in pull request #1383: HDDS-4194. Create a script to check AWS S3 compatibility

2020-09-23 Thread GitBox
elek commented on a change in pull request #1383: URL: https://github.com/apache/hadoop-ozone/pull/1383#discussion_r493495525 ## File path: hadoop-ozone/dist/src/main/smoketest/s3/s3_compatbility_check.sh ## @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Licensed to the Apache Softwa

[GitHub] [hadoop-ozone] elek commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-23 Thread GitBox
elek commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-697312216 > I think we should avoid logging chunk content. ContainerCommandRequestMessage implements related logic to clear data. I checked but it's not something which can be add

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-23 Thread GitBox
linyiqun commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r493463257 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java ## @@ -246,4 +246,9 @@ private OMConfigKeys() {

[GitHub] [hadoop-ozone] cku328 removed a comment on pull request #1368: HDDS-4156. add hierarchical layout to Chinese doc

2020-09-23 Thread GitBox
cku328 removed a comment on pull request #1368: URL: https://github.com/apache/hadoop-ozone/pull/1368#issuecomment-697277442 Overall LGTM. Thanks @flirmnave for working on this. This PR will make the Chinese docs more friendly. ---

[GitHub] [hadoop-ozone] cku328 commented on pull request #1368: HDDS-4156. add hierarchical layout to Chinese doc

2020-09-23 Thread GitBox
cku328 commented on pull request #1368: URL: https://github.com/apache/hadoop-ozone/pull/1368#issuecomment-697277442 Overall LGTM. Thanks @flirmnave for working on this. This PR will make the Chinese docs more friendly. Th

[GitHub] [hadoop-ozone] llemec commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
llemec commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-697250771 Hello @cxorm, Thank you for your review. I corrected the coding style as suggested. In a larger perspective, would it make sense to enhance checkstyle rules to warn fo

[GitHub] [hadoop-ozone] llemec commented on a change in pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
llemec commented on a change in pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#discussion_r493369935 ## File path: hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestInstanceHelper.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to t

[GitHub] [hadoop-ozone] llemec commented on a change in pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-23 Thread GitBox
llemec commented on a change in pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#discussion_r493369544 ## File path: hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestInstanceHelper.java ## @@ -0,0 +1,67 @@ +/** + * Licensed to t

[GitHub] [hadoop-ozone] maobaolong commented on pull request #1438: HDDS-4263. ReplicatiomManager shouldn't retain one healthy replica per origin node Id.

2020-09-23 Thread GitBox
maobaolong commented on pull request #1438: URL: https://github.com/apache/hadoop-ozone/pull/1438#issuecomment-697206664 Hi @nandakumar131 , thanks for the first review and suggestion, I agree to you that we should not remove the OriginDatanodeId handling blindly, now I add a condition for

<    4   5   6   7   8   9   10   11   12   13   >