[GitHub] [hadoop-ozone] captainzmc commented on pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-22 Thread GitBox
captainzmc commented on pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#issuecomment-697133328 Thanks @ChenSammi for the review. Review issues has been fixed. This is an automated message from the

[GitHub] [hadoop-ozone] captainzmc commented on pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
captainzmc commented on pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#issuecomment-697133030 Thanks @ChenSammi for the review. Review issues has been fixed. This is an automated message from the

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-22 Thread GitBox
adoroszlai commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-696662025 > > Thanks @elek for updating the patch. Interestingly log -v stopped working, even if I execute log first (as mentioned previously). > > I double-checked and it

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
fapifta commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492702681 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static boolean

[GitHub] [hadoop-ozone] fapifta commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-22 Thread GitBox
fapifta commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-696680680 Hello @llemec thank you for your comments and continued work on this PR. Indeed, I can accept the way based on your argument. +1(non-binding) to merge the

[GitHub] [hadoop-ozone] linyiqun commented on pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
linyiqun commented on pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#issuecomment-696588382 > > Hi @rakeshadr , some initial review comments below. > > In additional, one question from me: this is the first task of dir cache, there will be other further

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492877283 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static

[GitHub] [hadoop-ozone] cxorm commented on pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-09-22 Thread GitBox
cxorm commented on pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#issuecomment-696701388 Thanks @iamabug for the work, Overall looks great to me (including the fixes), I would committed it If all CI checks passed.

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
linyiqun commented on a change in pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#discussion_r492562181 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -2521,4 +2521,32 @@ filesystem semantics. + + +

[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr commented on pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#issuecomment-696600100 Thanks a lot @linyiqun for the useful review comments! This is an automated message from the Apache Git

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696839280 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [hadoop-ozone] github-actions[bot] closed pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

2020-09-22 Thread GitBox
github-actions[bot] closed pull request #1110: URL: https://github.com/apache/hadoop-ozone/pull/1110 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[jira] [Commented] (HDDS-541) Ozone Quota support.

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200538#comment-17200538 ] Rui Wang commented on HDDS-541: --- Thanks [~micahzhao]! I will start from the NameSpace jira. > Ozone Quota

[GitHub] [hadoop-ozone] avijayanhwx edited a comment on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx edited a comment on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696839280 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [hadoop-ozone] github-actions[bot] commented on pull request #1110: HDDS-3843. Throw the specific exception other than NPE.

2020-09-22 Thread GitBox
github-actions[bot] commented on pull request #1110: URL: https://github.com/apache/hadoop-ozone/pull/1110#issuecomment-696453432 Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.

[GitHub] [hadoop-ozone] fapifta commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
fapifta commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696692461 Hi @avijayanhwx the initial changes look good to me thank you for sharing the wip state, and it seems to be a good direction so far. Maybe one minor comment from

[GitHub] [hadoop-ozone] rakeshadr edited a comment on pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr edited a comment on pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#issuecomment-696600100 Thanks a lot @linyiqun for the useful review comments! Updated another commit addressing the comments.

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
linyiqun commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492798607 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static boolean

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
hanishakoneru commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492845659 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static

[GitHub] [hadoop-ozone] bshashikant commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-22 Thread GitBox
bshashikant commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r488543079 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr commented on a change in pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#discussion_r492480005 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java ## @@ -246,4 +246,15 @@ private OMConfigKeys() {

[jira] [Assigned] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-792: - Assignee: Rui Wang > Use md5 hash as ETag for Ozone S3 objects >

[jira] [Commented] (HDDS-792) Use md5 hash as ETag for Ozone S3 objects

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200509#comment-17200509 ] Rui Wang commented on HDDS-792: --- I can try to look at this one. > Use md5 hash as ETag for Ozone S3 objects

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492461120 ## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/BucketArgs.java ## @@ -66,17 +69,23 @@ * @param

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-22 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r492416925 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614416 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java ## @@ -72,5 +76,10 @@

[jira] [Commented] (HDDS-3208) Implement Ratis Snapshots on SCM

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200503#comment-17200503 ] Rui Wang commented on HDDS-3208: OzoneManager.installSnapshotFromLeader Download part:

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492526790 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java ## @@ -229,7 +229,9 @@ public String

[jira] [Comment Edited] (HDDS-541) Ozone Quota support.

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200489#comment-17200489 ] mingchao zhao edited comment on HDDS-541 at 9/23/20, 3:39 AM: -- Hi

[jira] [Assigned] (HDDS-3729) Support Namespace Level quota for Volume

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-3729: --- Assignee: Rui Wang (was: mingchao zhao) > Support Namespace Level quota for Volume >

[jira] [Assigned] (HDDS-3729) Support Namespace Level quota for Volume

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-3729: --- Assignee: Rui Wang > Support Namespace Level quota for Volume >

[jira] [Assigned] (HDDS-3729) Support Namespace Level quota for Volume

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-3729: --- Assignee: mingchao zhao > Support Namespace Level quota for Volume >

[jira] [Assigned] (HDDS-3729) Support Namespace Level quota for Volume

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-3729: --- Assignee: (was: Rui Wang) > Support Namespace Level quota for Volume >

[jira] [Assigned] (HDDS-3728) Support Storage space Level Quota for bucket

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-3728: --- Assignee: mingchao zhao (was: Rui Wang) > Support Storage space Level Quota for bucket >

[jira] [Comment Edited] (HDDS-541) Ozone Quota support.

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200489#comment-17200489 ] mingchao zhao edited comment on HDDS-541 at 9/23/20, 3:29 AM: -- Hi

[jira] [Commented] (HDDS-3728) Support Storage space Level Quota for bucket

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200492#comment-17200492 ] mingchao zhao commented on HDDS-3728: - [~amaliujia] I'm nearly finished with PR for space quota, you

[jira] [Commented] (HDDS-3730) Support Namespace Level quota for bucket

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200490#comment-17200490 ] mingchao zhao commented on HDDS-3730: - [~amaliujia] Yes, you can do this > Support Namespace Level

[jira] [Commented] (HDDS-541) Ozone Quota support.

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200489#comment-17200489 ] mingchao zhao commented on HDDS-541: Hi [~amaliujia], I would really welcome you to work on this JIRA

[jira] [Updated] (HDDS-4269) Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory

2020-09-22 Thread Wei-Chiu Chuang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDDS-4269: -- Labels: newbie (was: ) > Ozone DataNode thinks a volume is failed if an unexpected file is in

[jira] [Created] (HDDS-4269) Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory

2020-09-22 Thread Wei-Chiu Chuang (Jira)
Wei-Chiu Chuang created HDDS-4269: - Summary: Ozone DataNode thinks a volume is failed if an unexpected file is in the HDDS root directory Key: HDDS-4269 URL: https://issues.apache.org/jira/browse/HDDS-4269

[GitHub] [hadoop-ozone] avijayanhwx edited a comment on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx edited a comment on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-697011800 There are some follow up work items for the 'Prepare for Upgrade' flow. - Purging Logs after applying the last txn. - Validating that an OM is

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-697011800 There are some follow up work items for Prepare for Upgrade flow. - Purging Logs after applying the last txn. - Validating that an OM is "prepared for

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492976213 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static

[jira] [Assigned] (HDDS-4182) Onboard HDDS-3869 into Layout version management

2020-09-22 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4182: --- Assignee: (was: Aravindan Vijayan) > Onboard HDDS-3869 into Layout version

[GitHub] [hadoop-ozone] avijayanhwx edited a comment on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx edited a comment on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696839280 > Hi @avijayanhwx > > the initial changes look good to me thank you for sharing the wip state, and it seems to be a good direction so far. > Maybe one

[jira] [Assigned] (HDDS-3730) Support Namespace Level quota for bucket

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3730: -- Assignee: Rui Wang > Support Namespace Level quota for bucket >

[jira] [Commented] (HDDS-3728) Support Storage space Level Quota for bucket

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200212#comment-17200212 ] Rui Wang commented on HDDS-3728: [~simonss] [~micahzhao] do you mind me working on this JIRA? > Support

[jira] [Commented] (HDDS-3730) Support Namespace Level quota for bucket

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200213#comment-17200213 ] Rui Wang commented on HDDS-3730: [~simonss] [~micahzhao] do you mind me working on this JIRA?  This one

[jira] [Assigned] (HDDS-3728) Support Storage space Level Quota for bucket

2020-09-22 Thread Rui Wang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang reassigned HDDS-3728: -- Assignee: Rui Wang > Support Storage space Level Quota for bucket >

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696845211 > Only some minor comments above. > In additional, can you add the new unit test for PrepareForUpgrade scenario? Current unit test change cannot cover this.

[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696839280 > Hi @avijayanhwx > > the initial changes look good to me thank you for sharing the wip state, and it seems to be a good direction so far. > Maybe one minor

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492878138 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerStarter.java ## @@ -98,6 +98,28 @@ public void

[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
avijayanhwx commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492877283 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
hanishakoneru commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492845659 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static

[jira] [Assigned] (HDDS-4268) Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.

2020-09-22 Thread Aravindan Vijayan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan reassigned HDDS-4268: --- Assignee: Aravindan Vijayan > Prepare for Upgrade step should purge the log after

[jira] [Created] (HDDS-4268) Prepare for Upgrade step should purge the log after waiting for the last txn to be applied.

2020-09-22 Thread Aravindan Vijayan (Jira)
Aravindan Vijayan created HDDS-4268: --- Summary: Prepare for Upgrade step should purge the log after waiting for the last txn to be applied. Key: HDDS-4268 URL: https://issues.apache.org/jira/browse/HDDS-4268

[jira] [Updated] (HDDS-3297) TestOzoneClientKeyGenerator is flaky

2020-09-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3297: - Labels: TriagePending flaky-test ozone-flaky-test pull-request-available (was: TriagePending

[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #1442: HDDS-3297. Enable TestOzoneClientKeyGenerator.

2020-09-22 Thread GitBox
aryangupta1998 opened a new pull request #1442: URL: https://github.com/apache/hadoop-ozone/pull/1442 ## What changes were proposed in this pull request? Enable TestOzoneClientKeyGenerator. ## What is the link to the Apache JIRA

[jira] [Updated] (HDDS-3966) Intermittent crash in TestOMRatisSnapshots

2020-09-22 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3966: - Labels: pull-request-available (was: ) > Intermittent crash in TestOMRatisSnapshots >

[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #1441: HDDS-3966. Enable TestOMRatisSnapshots.

2020-09-22 Thread GitBox
aryangupta1998 opened a new pull request #1441: URL: https://github.com/apache/hadoop-ozone/pull/1441 ## What changes were proposed in this pull request? Enable TestOMRatisSnapshots. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3966

[jira] [Updated] (HDDS-541) Ozone Quota support.

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao updated HDDS-541: --- Description: Create a volume with just 1 MB as quota {code:java}

[jira] [Updated] (HDDS-541) Ozone Quota support.

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao updated HDDS-541: --- Attachment: (was: Ozone Quota Design.pdf) > Ozone Quota support. > > >

[jira] [Assigned] (HDDS-4106) Volume space: Supports clearing spaceQuota

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-4106: --- Assignee: mingchao zhao > Volume space: Supports clearing spaceQuota >

[jira] [Assigned] (HDDS-4105) Volume space: update spaceQuota should check the current quotaUsageInBytes

2020-09-22 Thread mingchao zhao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mingchao zhao reassigned HDDS-4105: --- Assignee: mingchao zhao > Volume space: update spaceQuota should check the current

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
linyiqun commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492801087 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -1179,15 +1229,22 @@ public void start()

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
linyiqun commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492798607 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static boolean

[jira] [Updated] (HDDS-4197) Failed to load existing service definition files: ...SubcommandWithParent

2020-09-22 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-4197: --- Labels: jdk11 (was: jdk11 pull-request-available) > Failed to load existing service

[GitHub] [hadoop-ozone] cxorm commented on pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-09-22 Thread GitBox
cxorm commented on pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#issuecomment-696701388 Thanks @iamabug for the work, Overall looks great to me (including the fixes), I would committed it If all CI checks passed.

[jira] [Created] (HDDS-4267) Ozone command always print warn message before execution

2020-09-22 Thread Yiqun Lin (Jira)
Yiqun Lin created HDDS-4267: --- Summary: Ozone command always print warn message before execution Key: HDDS-4267 URL: https://issues.apache.org/jira/browse/HDDS-4267 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
fapifta commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492706805 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManagerStarter.java ## @@ -98,6 +98,28 @@ public void

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
fapifta commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492705325 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -1179,15 +1229,22 @@ public void start()

[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
fapifta commented on a change in pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#discussion_r492702681 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -994,6 +1005,45 @@ public static boolean

[GitHub] [hadoop-ozone] fapifta commented on pull request #1430: HDDS-4227. Implement a 'Prepare For Upgrade' step in OM that applies all committed Ratis transactions.

2020-09-22 Thread GitBox
fapifta commented on pull request #1430: URL: https://github.com/apache/hadoop-ozone/pull/1430#issuecomment-696692461 Hi @avijayanhwx the initial changes look good to me thank you for sharing the wip state, and it seems to be a good direction so far. Maybe one minor comment from

[GitHub] [hadoop-ozone] fapifta commented on pull request #1425: HDDS-2981 Add unit tests for Proto [de]serialization

2020-09-22 Thread GitBox
fapifta commented on pull request #1425: URL: https://github.com/apache/hadoop-ozone/pull/1425#issuecomment-696680680 Hello @llemec thank you for your comments and continued work on this PR. Indeed, I can accept the way based on your argument. +1(non-binding) to merge the

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1272: HDDS-2660. Create insight point for datanode container protocol

2020-09-22 Thread GitBox
adoroszlai commented on pull request #1272: URL: https://github.com/apache/hadoop-ozone/pull/1272#issuecomment-696662025 > > Thanks @elek for updating the patch. Interestingly log -v stopped working, even if I execute log first (as mentioned previously). > > I double-checked and it

[jira] [Created] (HDDS-4266) CreateFile : store parent dir entries into DirTable and file entry into separate FileTable

2020-09-22 Thread Rakesh Radhakrishnan (Jira)
Rakesh Radhakrishnan created HDDS-4266: -- Summary: CreateFile : store parent dir entries into DirTable and file entry into separate FileTable Key: HDDS-4266 URL:

[jira] [Updated] (HDDS-4222) [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread Rakesh Radhakrishnan (Jira)
[ https://issues.apache.org/jira/browse/HDDS-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh Radhakrishnan updated HDDS-4222: --- Status: Patch Available (was: Open) > [OzoneFS optimization] Provide a mechanism for

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614881 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java ## @@ -99,6 +103,11 @@

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614881 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCreateResponse.java ## @@ -99,6 +103,11 @@

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614631 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyCommitResponse.java ## @@ -77,6 +80,11 @@

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1431: HDDS-4254. Bucket space: add usedBytes and update it when create and delete key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1431: URL: https://github.com/apache/hadoop-ozone/pull/1431#discussion_r492614416 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMAllocateBlockResponse.java ## @@ -72,5 +76,10 @@

[GitHub] [hadoop-ozone] runzhiwang commented on a change in pull request #1371: HDDS-2922. Balance ratis leader distribution in datanodes

2020-09-22 Thread GitBox
runzhiwang commented on a change in pull request #1371: URL: https://github.com/apache/hadoop-ozone/pull/1371#discussion_r492595294 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java ## @@ -98,8 +105,65 @@ private

[GitHub] [hadoop-ozone] rakeshadr edited a comment on pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr edited a comment on pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#issuecomment-696600100 Thanks a lot @linyiqun for the useful review comments! Updated another commit addressing the comments.

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr commented on a change in pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#discussion_r492582953 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -2521,4 +2521,32 @@ filesystem semantics. + + +

[GitHub] [hadoop-ozone] rakeshadr commented on pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr commented on pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#issuecomment-696600100 Thanks a lot @linyiqun for the useful review comments! This is an automated message from the Apache Git

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
rakeshadr commented on a change in pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#discussion_r492582620 ## File path: hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/cache/TestOMMetadataCache.java ## @@ -0,0 +1,276 @@ +/** + *

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492575126 ## File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java ## @@ -707,6 +707,62

[GitHub] [hadoop-ozone] linyiqun commented on pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
linyiqun commented on pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#issuecomment-696588382 > > Hi @rakeshadr , some initial review comments below. > > In additional, one question from me: this is the first task of dir cache, there will be other further

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1437: HDDS-4222: [OzoneFS optimization] Provide a mechanism for efficient path lookup

2020-09-22 Thread GitBox
linyiqun commented on a change in pull request #1437: URL: https://github.com/apache/hadoop-ozone/pull/1437#discussion_r492562181 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -2521,4 +2521,32 @@ filesystem semantics. + + +

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492533073 ## File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto ## @@ -314,6 +314,8 @@ enum Status { PARTIAL_RENAME = 65;

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492526790 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java ## @@ -229,7 +229,9 @@ public String

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1434: HDDS-3727. Volume space: check quotaUsageInBytes when write key.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1434: URL: https://github.com/apache/hadoop-ozone/pull/1434#discussion_r492526790 ## File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java ## @@ -229,7 +229,9 @@ public String

[jira] [Created] (HDDS-4265) Refactor OzoneQuota to make it easy to support more quota type

2020-09-22 Thread Sammi Chen (Jira)
Sammi Chen created HDDS-4265: Summary: Refactor OzoneQuota to make it easy to support more quota type Key: HDDS-4265 URL: https://issues.apache.org/jira/browse/HDDS-4265 Project: Hadoop Distributed Data

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492517940 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java ## @@ -297,4 +300,17

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492518254 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java ## @@ -150,6

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492517940 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java ## @@ -297,4 +300,17

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492515786 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/volume/SetQuotaHandler.java ## @@ -0,0 +1,71 @@ +/* + * Licensed to

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492515197 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/volume/SetQuotaHandler.java ## @@ -0,0 +1,71 @@ +/* + * Licensed to

[jira] [Created] (HDDS-4264) Uniform naming conventions of Ozone Shell Options.

2020-09-22 Thread mingchao zhao (Jira)
mingchao zhao created HDDS-4264: --- Summary: Uniform naming conventions of Ozone Shell Options. Key: HDDS-4264 URL: https://issues.apache.org/jira/browse/HDDS-4264 Project: Hadoop Distributed Data Store

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1412: HDDS-3751. Ozone sh client support bucket quota option.

2020-09-22 Thread GitBox
ChenSammi commented on a change in pull request #1412: URL: https://github.com/apache/hadoop-ozone/pull/1412#discussion_r492511397 ## File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/ClearSpaceQuotaOptions.java ## @@ -0,0 +1,35 @@ +/* + * Licensed to

  1   2   >