[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1104: HDDS-3612. Allow mounting bucket under other volume

2020-07-09 Thread GitBox
adoroszlai commented on a change in pull request #1104: URL: https://github.com/apache/hadoop-ozone/pull/1104#discussion_r452020792 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java ## @@ -2145,37 +2168,51 @@ public OmKeyLocati

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1162: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen…

2020-07-09 Thread GitBox
ChenSammi commented on pull request #1162: URL: https://github.com/apache/hadoop-ozone/pull/1162#issuecomment-655969692 > Thanks @ChenSammi for reporting the issue and propose the fix. The change LGTM. > > I have a question wrt the excess calculation in handleOverReplica

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1162: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen…

2020-07-09 Thread GitBox
ChenSammi commented on a change in pull request #1162: URL: https://github.com/apache/hadoop-ozone/pull/1162#discussion_r452035232 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java ## @@ -543,11 +543,15 @@ private v

[GitHub] [hadoop-ozone] prashantpogde opened a new pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.

2020-07-09 Thread GitBox
prashantpogde opened a new pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182 ## What changes were proposed in this pull request? Ozone Manager Token Identifier table should use in-house serialization rather than rely on proto serialization for key. ## W

[jira] [Updated] (HDDS-3926) Ozone Manager Token Identifier table should use in-house serialization rather than rely on proto serialization for key.

2020-07-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3926: - Labels: pull-request-available upgrade-p0 (was: upgrade-p0) > Ozone Manager Token Identifier tabl

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1162: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen…

2020-07-09 Thread GitBox
ChenSammi commented on pull request #1162: URL: https://github.com/apache/hadoop-ozone/pull/1162#issuecomment-655983803 > Thanks for this change. I also wonder if there is a bug in the method `isContainerUnderReplicated(...)` which leads to this problem, and results in more processing that

[jira] [Created] (HDDS-3942) Refine isContainerUnderReplicated check by including inflight replications

2020-07-09 Thread Sammi Chen (Jira)
Sammi Chen created HDDS-3942: Summary: Refine isContainerUnderReplicated check by including inflight replications Key: HDDS-3942 URL: https://issues.apache.org/jira/browse/HDDS-3942 Project: Hadoop Distri

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1162: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen…

2020-07-09 Thread GitBox
ChenSammi commented on a change in pull request #1162: URL: https://github.com/apache/hadoop-ozone/pull/1162#discussion_r452052352 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java ## @@ -543,11 +543,15 @@ private v

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1163: HDDS-3920. Too many redudant replications due to fail to get node's a…

2020-07-09 Thread GitBox
ChenSammi commented on a change in pull request #1163: URL: https://github.com/apache/hadoop-ozone/pull/1163#discussion_r452056907 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java ## @@ -260,12 +260,21 @@ public RegisteredC

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1163: HDDS-3920. Too many redudant replications due to fail to get node's a…

2020-07-09 Thread GitBox
ChenSammi commented on a change in pull request #1163: URL: https://github.com/apache/hadoop-ozone/pull/1163#discussion_r452061030 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java ## @@ -260,12 +260,21 @@ public RegisteredC

[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1163: HDDS-3920. Too many redudant replications due to fail to get node's a…

2020-07-09 Thread GitBox
ChenSammi commented on a change in pull request #1163: URL: https://github.com/apache/hadoop-ozone/pull/1163#discussion_r452064097 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReportHandler.java ## @@ -102,8 +102,15 @@ public

[jira] [Created] (HDDS-3943) Cleanup empty container directory

2020-07-09 Thread Sammi Chen (Jira)
Sammi Chen created HDDS-3943: Summary: Cleanup empty container directory Key: HDDS-3943 URL: https://issues.apache.org/jira/browse/HDDS-3943 Project: Hadoop Distributed Data Store Issue Type: Imp

[jira] [Updated] (HDDS-3943) Cleanup empty container directory

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3943: - Description: Here is the log after datanode restart. One thing actullay I'd like to know is why there are

[jira] [Updated] (HDDS-3943) Cleanup empty container directory

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3943: - Description: Here is the log after datanode restart. One thing actullay I'd like to know is why these are

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
adoroszlai commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452098749 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java ## @@ -116,89 +111,112 @@

[GitHub] [hadoop-ozone] sodonnel merged pull request #1156: HDDS-3879. Introduce SCM and OM layoutVersion zero to the VERSION file

2020-07-09 Thread GitBox
sodonnel merged pull request #1156: URL: https://github.com/apache/hadoop-ozone/pull/1156 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Commented] (HDDS-3830) Introduce OM layout version 'v0'.

2020-07-09 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154408#comment-17154408 ] Stephen O'Donnell commented on HDDS-3830: - This is covered by the change committed

[jira] [Resolved] (HDDS-3879) Introduce SCM layout version 'v0'.

2020-07-09 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3879. - Fix Version/s: 0.6.0 Resolution: Fixed > Introduce SCM layout version 'v0'. > -

[jira] [Resolved] (HDDS-3830) Introduce OM layout version 'v0'.

2020-07-09 Thread Stephen O'Donnell (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDDS-3830. - Resolution: Duplicate > Introduce OM layout version 'v0'. > --

[GitHub] [hadoop-ozone] runitao commented on pull request #1136: HDDS-3798. Collect more info about DN in recon web

2020-07-09 Thread GitBox
runitao commented on pull request #1136: URL: https://github.com/apache/hadoop-ozone/pull/1136#issuecomment-656039299 ``` [ERROR] Failures: [ERROR] TestBlockOutputStreamFlushDelay.testFullBufferCondition:582 expected:<11> but was:<13> ``` I think this failure case is unrelate

[jira] [Updated] (HDDS-3223) Improve s3g read 1GB object efficiency by 100 times

2020-07-09 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3223: - Summary: Improve s3g read 1GB object efficiency by 100 times (was: Improve s3g read 1GB object efficienc

[jira] [Commented] (HDDS-3852) Failed to import replicated container

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154442#comment-17154442 ] Sammi Chen commented on HDDS-3852: -- Don't have more information since the downloaded cont

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1147: HDDS-3892. Datanode initialization is too slow when there are thousan…

2020-07-09 Thread GitBox
ChenSammi commented on pull request #1147: URL: https://github.com/apache/hadoop-ozone/pull/1147#issuecomment-656074756 Thanks @bharatviswa504 and @adoroszlai for the review. lock per containerDBPath in ContainerCache is a good idea. I tried it but cannot solve our problem given a volume

[jira] [Created] (HDDS-3944) OM StateMachine unpause fails with NPE

2020-07-09 Thread Attila Doroszlai (Jira)
Attila Doroszlai created HDDS-3944: -- Summary: OM StateMachine unpause fails with NPE Key: HDDS-3944 URL: https://issues.apache.org/jira/browse/HDDS-3944 Project: Hadoop Distributed Data Store

[jira] [Created] (HDDS-3945) ContainerReplicaNotFoundException when remove a replica in ContainerReportHandler

2020-07-09 Thread Sammi Chen (Jira)
Sammi Chen created HDDS-3945: Summary: ContainerReplicaNotFoundException when remove a replica in ContainerReportHandler Key: HDDS-3945 URL: https://issues.apache.org/jira/browse/HDDS-3945 Project: Hadoop

[jira] [Created] (HDDS-3946) s3g multi-upload failed with partName check

2020-07-09 Thread Sammi Chen (Jira)
Sammi Chen created HDDS-3946: Summary: s3g multi-upload failed with partName check Key: HDDS-3946 URL: https://issues.apache.org/jira/browse/HDDS-3946 Project: Hadoop Distributed Data Store Issue

[jira] [Commented] (HDDS-3946) s3g multi-upload failed with partName check

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154499#comment-17154499 ] Sammi Chen commented on HDDS-3946: -- Hi [~bharat], do you think we can loose the partName

[jira] [Updated] (HDDS-3946) s3g multi-upload failed with partName check

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3946: - Description: LOGs in S3g, INVALID_PART org.apache.hadoop.ozone.om.exceptions.OMException: Complete Multi

[jira] [Created] (HDDS-3947) Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs

2020-07-09 Thread Rakesh Radhakrishnan (Jira)
Rakesh Radhakrishnan created HDDS-3947: -- Summary: Sort DNs for client when the key is a file for #getFileStatus #listStatus APIs Key: HDDS-3947 URL: https://issues.apache.org/jira/browse/HDDS-3947

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1164: HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK

2020-07-09 Thread GitBox
rakeshadr commented on a change in pull request #1164: URL: https://github.com/apache/hadoop-ozone/pull/1164#discussion_r452196471 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java ## @@ -1706,36 +1718,41 @@ public OzoneFileS

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1174: HDDS-3918. ConcurrentModificationException in ContainerReportHandler.…

2020-07-09 Thread GitBox
adoroszlai commented on a change in pull request #1174: URL: https://github.com/apache/hadoop-ozone/pull/1174#discussion_r452179572 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/NodeStateMap.java ## @@ -88,7 +88,7 @@ public void addN

[GitHub] [hadoop-ozone] rakeshadr commented on a change in pull request #1164: HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK

2020-07-09 Thread GitBox
rakeshadr commented on a change in pull request #1164: URL: https://github.com/apache/hadoop-ozone/pull/1164#discussion_r452196471 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java ## @@ -1706,36 +1718,41 @@ public OzoneFileS

[GitHub] [hadoop-ozone] elek commented on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
elek commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656112630 > We want to support client-less config of HA eventually (using DNS for example). In that case we cannot look at the client side configs to know how many clusters/services the

[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1183: HDDS-3944. OM StateMachine unpause fails with NPE

2020-07-09 Thread GitBox
adoroszlai opened a new pull request #1183: URL: https://github.com/apache/hadoop-ozone/pull/1183 ## What changes were proposed in this pull request? Add missing `setIndexToTerm(this::getTermForIndex)` call to `OzoneManagerDoubleBuffer.Builder` chain, and extract duplicate code to a

[jira] [Updated] (HDDS-3944) OM StateMachine unpause fails with NPE

2020-07-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3944: - Labels: pull-request-available (was: ) > OM StateMachine unpause fails with NPE > ---

[GitHub] [hadoop-ozone] iamabug commented on a change in pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-07-09 Thread GitBox
iamabug commented on a change in pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#discussion_r452206180 ## File path: hadoop-hdds/docs/content/security/SecuringDatanodes.zh.md ## @@ -0,0 +1,53 @@ +--- +title: "安全化 Datanode" +date: "2019-April-03" +weigh

[GitHub] [hadoop-ozone] iamabug commented on pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-07-09 Thread GitBox
iamabug commented on pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#issuecomment-656126947 > > @xiaoyuyao @cxorm Could you help review this ? Thanks. BTW, I cannot find request review button on this page. > > You can do that on the top right, "reviewers" bu

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1081: HDDS-3809. Make number of open containers on a datanode a function of no of volumes reported by it.

2020-07-09 Thread GitBox
adoroszlai commented on a change in pull request #1081: URL: https://github.com/apache/hadoop-ozone/pull/1081#discussion_r452326354 ## File path: hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java ## @@ -605,6 +616,12 @@ public void

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1174: HDDS-3918. ConcurrentModificationException in ContainerReportHandler.…

2020-07-09 Thread GitBox
xiaoyuyao commented on a change in pull request #1174: URL: https://github.com/apache/hadoop-ozone/pull/1174#discussion_r452326595 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/NodeStateMap.java ## @@ -88,7 +88,7 @@ public void addNo

[GitHub] [hadoop-ozone] adoroszlai merged pull request #1051: HDDS-3747. Redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread GitBox
adoroszlai merged pull request #1051: URL: https://github.com/apache/hadoop-ozone/pull/1051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[jira] [Assigned] (HDDS-3918) ConcurrentModificationException in ContainerReportHandler.onMessage

2020-07-09 Thread Xiaoyu Yao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao reassigned HDDS-3918: Assignee: Xiaoyu Yao (was: Nanda kumar) > ConcurrentModificationException in ContainerReportHandle

[jira] [Updated] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-3747: --- Labels: (was: pull-request-available) > Remove the redundancy if condition code in ListPipel

[jira] [Resolved] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai resolved HDDS-3747. Fix Version/s: 0.7.0 Resolution: Done > Remove the redundancy if condition code in Li

[jira] [Updated] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-3747: --- Issue Type: Improvement (was: New Feature) > Remove the redundancy if condition code in ListP

[jira] [Commented] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154705#comment-17154705 ] Attila Doroszlai commented on HDDS-3747: [~arp] can you please add [~hemanthboyina

[jira] [Updated] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-3747: --- Affects Version/s: (was: 0.7.0) > Remove the redundancy if condition code in ListPipelines

[GitHub] [hadoop-ozone] arp7 commented on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
arp7 commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656221254 Yes one risk is when people code up their apps or scripts to use implicit paths. Then we add a second cluster and now their paths that used to work previously don't work anymo

[GitHub] [hadoop-ozone] arp7 edited a comment on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
arp7 edited a comment on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656221254 Yes one risk is when people code up their apps or scripts to use implicit paths. Then we add a second cluster and now their paths that used to work previously don't wor

[jira] [Commented] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Arpit Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154712#comment-17154712 ] Arpit Agarwal commented on HDDS-3747: - Done. > Remove the redundancy if condition cod

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
bharatviswa504 commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656224008 As I understood with defaultFs it's always possible to use implicit defaults. Do you see any risk to use default cluster if only one is defined? Are you referr

[GitHub] [hadoop-ozone] bharatviswa504 edited a comment on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
bharatviswa504 edited a comment on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656224008 As I understood with defaultFs it's always possible to use implicit defaults. Do you see any risk to use default cluster if only one is defined? Are you

[jira] [Assigned] (HDDS-3747) Remove the redundancy if condition code in ListPipelinesSubcommand

2020-07-09 Thread Arpit Agarwal (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal reassigned HDDS-3747: --- Assignee: HemanthBoyina > Remove the redundancy if condition code in ListPipelinesSubcommand

[GitHub] [hadoop-ozone] arp7 commented on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
arp7 commented on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656230263 So iiuc the main convenience argument is that we can type paths like `/user/bob/foo` instead of `o3fs://bucket.volume.om/user/bob/foo` right? I think some of the overhea

[GitHub] [hadoop-ozone] arp7 edited a comment on pull request #1149: HDDS-3878. Make OMHA serviceID optional if one (but only one) is defined in the config

2020-07-09 Thread GitBox
arp7 edited a comment on pull request #1149: URL: https://github.com/apache/hadoop-ozone/pull/1149#issuecomment-656230263 So iiuc the main convenience argument is that we can type paths like `/user/bob/foo` instead of `o3fs://bucket.volume.om/user/bob/foo` right? With ofs, buckets an

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #1166: HDDS-3914. Remove LevelDB configuration option for DN Metastore

2020-07-09 Thread GitBox
hanishakoneru commented on a change in pull request #1166: URL: https://github.com/apache/hadoop-ozone/pull/1166#discussion_r452391946 ## File path: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java ## @@ -186,8

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #1166: HDDS-3914. Remove LevelDB configuration option for DN Metastore

2020-07-09 Thread GitBox
hanishakoneru commented on a change in pull request #1166: URL: https://github.com/apache/hadoop-ozone/pull/1166#discussion_r452392038 ## File path: hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java ## @@ -90,12 +93,12 @@ public void init(

[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request #1166: HDDS-3914. Remove LevelDB configuration option for DN Metastore

2020-07-09 Thread GitBox
hanishakoneru commented on a change in pull request #1166: URL: https://github.com/apache/hadoop-ozone/pull/1166#discussion_r452392147 ## File path: hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java ## @@ -166,12 +170,12 @@ public void tes

[jira] [Assigned] (HDDS-3925) SCM Pipeline DB should directly use UUID bytes for key rather than rely on proto serialization for key.

2020-07-09 Thread Siddharth Wagle (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle reassigned HDDS-3925: - Assignee: Prashant Pogde (was: Istvan Fajth) > SCM Pipeline DB should directly use UUID

[jira] [Commented] (HDDS-3925) SCM Pipeline DB should directly use UUID bytes for key rather than rely on proto serialization for key.

2020-07-09 Thread Istvan Fajth (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154884#comment-17154884 ] Istvan Fajth commented on HDDS-3925: Ok, so the problem here is the following: - in SC

[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #1181: [POC] HDDS-3915. Simple trash emptier on OM

2020-07-09 Thread GitBox
smengcl commented on a change in pull request #1181: URL: https://github.com/apache/hadoop-ozone/pull/1181#discussion_r452464751 ## File path: hadoop-ozone/dist/src/main/compose/ozone/docker-config ## @@ -14,6 +14,9 @@ # See the License for the specific language governing perm

[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #1181: [POC] HDDS-3915. Simple trash emptier on OM

2020-07-09 Thread GitBox
smengcl commented on a change in pull request #1181: URL: https://github.com/apache/hadoop-ozone/pull/1181#discussion_r452464751 ## File path: hadoop-ozone/dist/src/main/compose/ozone/docker-config ## @@ -14,6 +14,9 @@ # See the License for the specific language governing perm

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
xiaoyuyao commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452474846 ## File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto ## @@ -867,10 +867,10 @@ message DeletedKeys { } message Dele

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
bharatviswa504 commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452484762 ## File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto ## @@ -867,10 +867,10 @@ message DeletedKeys { } message

[GitHub] [hadoop-ozone] codecov-commenter commented on pull request #1166: HDDS-3914. Remove LevelDB configuration option for DN Metastore

2020-07-09 Thread GitBox
codecov-commenter commented on pull request #1166: URL: https://github.com/apache/hadoop-ozone/pull/1166#issuecomment-656347397 # [Codecov](https://codecov.io/gh/apache/hadoop-ozone/pull/1166?src=pr&el=h1) Report > Merging [#1166](https://codecov.io/gh/apache/hadoop-ozone/pull/1166?src

[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-07-09 Thread GitBox
smengcl commented on a change in pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#discussion_r452504389 ## File path: hadoop-hdds/docs/content/security/SecuringDatanodes.zh.md ## @@ -0,0 +1,53 @@ +--- +title: "安全化 Datanode" +date: "2019-April-03" +weigh

[GitHub] [hadoop-ozone] smengcl commented on a change in pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-07-09 Thread GitBox
smengcl commented on a change in pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#discussion_r452504389 ## File path: hadoop-hdds/docs/content/security/SecuringDatanodes.zh.md ## @@ -0,0 +1,53 @@ +--- +title: "安全化 Datanode" +date: "2019-April-03" +weigh

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.

2020-07-09 Thread GitBox
xiaoyuyao commented on a change in pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182#discussion_r452533512 ## File path: hadoop-hdds/interface-client/src/main/proto/proto.lock ## @@ -1476,6 +1476,21 @@ } ], "messages": [ +

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.

2020-07-09 Thread GitBox
xiaoyuyao commented on a change in pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182#discussion_r452533512 ## File path: hadoop-hdds/interface-client/src/main/proto/proto.lock ## @@ -1476,6 +1476,21 @@ } ], "messages": [ +

[GitHub] [hadoop-ozone] iamabug commented on pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-07-09 Thread GitBox
iamabug commented on pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#issuecomment-656391585 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on

[jira] [Commented] (HDDS-3363) Intermittent failure in testContainerImportExport

2020-07-09 Thread Xiaoyu Yao (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154976#comment-17154976 ] Xiaoyu Yao commented on HDDS-3363: -- Hit a similar issue in recent run:https://github.com

[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1162: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen…

2020-07-09 Thread GitBox
xiaoyuyao commented on a change in pull request #1162: URL: https://github.com/apache/hadoop-ozone/pull/1162#discussion_r452541315 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ReplicationManager.java ## @@ -543,11 +543,15 @@ private v

[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1183: HDDS-3944. OM StateMachine unpause fails with NPE

2020-07-09 Thread GitBox
bharatviswa504 commented on pull request #1183: URL: https://github.com/apache/hadoop-ozone/pull/1183#issuecomment-656398721 Thank You @adoroszlai for the contribution and @hanishakoneru for the review. This is an automated m

[GitHub] [hadoop-ozone] bharatviswa504 merged pull request #1183: HDDS-3944. OM StateMachine unpause fails with NPE

2020-07-09 Thread GitBox
bharatviswa504 merged pull request #1183: URL: https://github.com/apache/hadoop-ozone/pull/1183 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

[jira] [Resolved] (HDDS-3944) OM StateMachine unpause fails with NPE

2020-07-09 Thread Bharat Viswanadham (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-3944. -- Fix Version/s: 0.6.0 Resolution: Fixed > OM StateMachine unpause fails with NPE >

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
bharatviswa504 commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452559759 ## File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto ## @@ -867,10 +867,10 @@ message DeletedKeys { } message

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
bharatviswa504 commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452560417 ## File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto ## @@ -867,10 +867,10 @@ message DeletedKeys { } message

[jira] [Created] (HDDS-3948) Release/reacquire lock for OMKeysDeleteRequest

2020-07-09 Thread Bharat Viswanadham (Jira)
Bharat Viswanadham created HDDS-3948: Summary: Release/reacquire lock for OMKeysDeleteRequest Key: HDDS-3948 URL: https://issues.apache.org/jira/browse/HDDS-3948 Project: Hadoop Distributed Data St

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
bharatviswa504 commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452561712 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java ## @@ -116,89 +111,112

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
bharatviswa504 commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452561797 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java ## @@ -116,93 +111,116

[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1169: HDDS-3930. Fix OMKeyDeletesRequest.

2020-07-09 Thread GitBox
bharatviswa504 commented on a change in pull request #1169: URL: https://github.com/apache/hadoop-ozone/pull/1169#discussion_r452561831 ## File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java ## @@ -116,93 +111,116

[GitHub] [hadoop-ozone] iamabug commented on a change in pull request #1175: HDDS-2766. security/SecuringDataNodes.md

2020-07-09 Thread GitBox
iamabug commented on a change in pull request #1175: URL: https://github.com/apache/hadoop-ozone/pull/1175#discussion_r452569796 ## File path: hadoop-hdds/docs/content/security/SecuringDatanodes.zh.md ## @@ -0,0 +1,53 @@ +--- +title: "安全化 Datanode" +date: "2019-April-03" +weigh

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.

2020-07-09 Thread GitBox
prashantpogde commented on a change in pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182#discussion_r452573445 ## File path: hadoop-hdds/interface-client/src/main/proto/proto.lock ## @@ -1476,6 +1476,21 @@ } ], "messages": [

[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1182: HDDS-3926. OM Token Identifier table should use in-house serialization.

2020-07-09 Thread GitBox
prashantpogde commented on a change in pull request #1182: URL: https://github.com/apache/hadoop-ozone/pull/1182#discussion_r452573311 ## File path: hadoop-hdds/interface-client/src/main/proto/proto.lock ## @@ -1476,6 +1476,21 @@ } ], "messages": [

[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1162: HDDS-3921. IllegalArgumentException triggered in SCMContainerPlacemen…

2020-07-09 Thread GitBox
ChenSammi commented on pull request #1162: URL: https://github.com/apache/hadoop-ozone/pull/1162#issuecomment-656443918 testDeleteKeyWithSlowFollower failed at leader membership check step. The test passed locally. It seems a timing issue, not relevant to this patch.

[jira] [Updated] (HDDS-3933) Fix memory leak because of too many Datanode State Machine Thread

2020-07-09 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3933: - Summary: Fix memory leak because of too many Datanode State Machine Thread (was: memory leak because of t

[jira] [Commented] (HDDS-3916) Change the return type in DeleteKeys and renameKeys in om client side.

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155059#comment-17155059 ] Sammi Chen commented on HDDS-3916: -- Hi [~micahzhao],will this break the API level backwor

[jira] [Commented] (HDDS-3912) Change SCM ContainerDB key to proto structure to support backward compatibility.

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17155061#comment-17155061 ] Sammi Chen commented on HDDS-3912: -- Hi [~pifta] and [~avijayan], do we need to put this i

[jira] [Created] (HDDS-3949) Flaky test cases

2020-07-09 Thread Sammi Chen (Jira)
Sammi Chen created HDDS-3949: Summary: Flaky test cases Key: HDDS-3949 URL: https://issues.apache.org/jira/browse/HDDS-3949 Project: Hadoop Distributed Data Store Issue Type: New Feature

[jira] [Resolved] (HDDS-3949) Flaky test cases

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen resolved HDDS-3949. -- Resolution: Duplicate > Flaky test cases > > > Key: HDDS-3949 >

[jira] [Updated] (HDDS-3938) Flaky TestWatchForCommit#test2WayCommitForTimeoutException

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3938: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > Flaky TestWatchForCommit#test2WayCommitForTim

[jira] [Updated] (HDDS-3938) Flaky TestWatchForCommit#test2WayCommitForTimeoutException

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3938: - Labels: 0.7.0 (was: ) > Flaky TestWatchForCommit#test2WayCommitForTimeoutException >

[jira] [Updated] (HDDS-3873) Intermittent failure in Recon acceptance test due to too many pipelines

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3873: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > Intermittent failure in Recon acceptance test

[jira] [Updated] (HDDS-3874) ITestRootedOzoneContract tests are flaky

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3874: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > ITestRootedOzoneContract tests are flaky > --

[GitHub] [hadoop-ozone] iamabug opened a new pull request #1184: HDDS-2767. security/SecuringTDE.md

2020-07-09 Thread GitBox
iamabug opened a new pull request #1184: URL: https://github.com/apache/hadoop-ozone/pull/1184 ## What changes were proposed in this pull request? translation to https://hadoop.apache.org/ozone/docs/0.5.0-beta/security/securingtde.html ## What is the link to the Apache JIRA

[jira] [Updated] (HDDS-2767) security/SecuringTDE.md

2020-07-09 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2767: - Labels: pull-request-available (was: ) > security/SecuringTDE.md > --- > >

[jira] [Updated] (HDDS-3762) Intermittent failure in TestDeleteWithSlowFollower

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3762: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > Intermittent failure in TestDeleteWithSlowFol

[jira] [Updated] (HDDS-3516) Intermittent failure in TestReadRetries

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3516: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > Intermittent failure in TestReadRetries > ---

[jira] [Updated] (HDDS-3506) TestOzoneFileInterfaces is flaky

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3506: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > TestOzoneFileInterfaces is flaky > --

[jira] [Updated] (HDDS-3363) Intermittent failure in testContainerImportExport

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3363: - Parent: HDDS-1127 Issue Type: Sub-task (was: Bug) > Intermittent failure in testContainerImportEx

[jira] [Updated] (HDDS-3358) Intermittent test failure related to a race conditon during PipelineManager close

2020-07-09 Thread Sammi Chen (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sammi Chen updated HDDS-3358: - Parent: HDDS-1127 Issue Type: Sub-task (was: Improvement) > Intermittent test failure related to

  1   2   >