[jira] [Commented] (HDDS-3294) Flaky test TestContainerStateMachineFailureOnRead#testReadStateMachineFailureClosesPipeline

2020-06-05 Thread Siyao Meng (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126460#comment-17126460 ] Siyao Meng commented on HDDS-3294: -- Note this test class is disabled (ignored) now. > Fl

[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume

2020-06-05 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-3612: --- Labels: Triaged (was: Triaged pull-request-available) > Allow mounting bucket under other vol

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1009: HDDS-3612. Document details of bucket mount design

2020-06-05 Thread GitBox
adoroszlai commented on pull request #1009: URL: https://github.com/apache/hadoop-ozone/pull/1009#issuecomment-639306784 Thanks @arp7 for reviewing this doc. I have updated it with some clarifications. Also changed to use "symbolic link" terminology instead of "mount" for consistency. T

[jira] [Updated] (HDDS-3612) Allow mounting bucket under other volume

2020-06-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3612: - Labels: Triaged pull-request-available (was: Triaged) > Allow mounting bucket under other volume

[GitHub] [hadoop-ozone] adoroszlai edited a comment on pull request #1009: HDDS-3612. Document details of bucket mount design

2020-06-05 Thread GitBox
adoroszlai edited a comment on pull request #1009: URL: https://github.com/apache/hadoop-ozone/pull/1009#issuecomment-639306784 Thanks @arp7 for reviewing this doc. * I have updated it with some clarifications based on your comments. * Changed permission required for following the

[jira] [Resolved] (HDDS-3710) Merge archived jacoco coverage results

2020-06-05 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3710. --- Fix Version/s: 0.6.0 Resolution: Fixed > Merge archived jacoco coverage results > -

[GitHub] [hadoop-ozone] elek merged pull request #1014: HDDS-3710. Merge archived jacoco coverage results

2020-06-05 Thread GitBox
elek merged pull request #1014: URL: https://github.com/apache/hadoop-ozone/pull/1014 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hadoop-ozone] elek commented on pull request #1014: HDDS-3710. Merge archived jacoco coverage results

2020-06-05 Thread GitBox
elek commented on pull request #1014: URL: https://github.com/apache/hadoop-ozone/pull/1014#issuecomment-639322368 Thanks @vivekratnavel the review. I am merging it now. Sonar master will be updated with coverage data after the next full green build. --

[GitHub] [hadoop-ozone] adoroszlai opened a new pull request #1024: HDDS-3720. Datanode configuration object has wrong values

2020-06-05 Thread GitBox
adoroszlai opened a new pull request #1024: URL: https://github.com/apache/hadoop-ozone/pull/1024 ## What changes were proposed in this pull request? 1. `@Config` annotation was assigned to the wrong fields (the ones with default values) in `DatanodeConfiguration` - fixed. 2. Adde

[jira] [Updated] (HDDS-3720) Datanode configuration object has wrong values

2020-06-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3720: - Labels: Triaged pull-request-available (was: Triaged) > Datanode configuration object has wrong v

[GitHub] [hadoop-ozone] elek merged pull request #1018: HDDS-3716. Compile of Ozone fails with JDK 11+

2020-06-05 Thread GitBox
elek merged pull request #1018: URL: https://github.com/apache/hadoop-ozone/pull/1018 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[GitHub] [hadoop-ozone] elek commented on pull request #1018: HDDS-3716. Compile of Ozone fails with JDK 11+

2020-06-05 Thread GitBox
elek commented on pull request #1018: URL: https://github.com/apache/hadoop-ozone/pull/1018#issuecomment-639327547 Thanks the review @arp7 and @adoroszlai (and the report @bshashikant). I am merging it now. BTW eventually we can switch to use java11 to compile (I am talking about th

[jira] [Resolved] (HDDS-3716) Compile of Ozone fails with JDK 11+

2020-06-05 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek resolved HDDS-3716. --- Fix Version/s: 0.6.0 Resolution: Fixed > Compile of Ozone fails with JDK 11+ >

[GitHub] [hadoop-ozone] elek merged pull request #949: HDDS-3621. Separate client/server/admin proto files of HDDS to separated subprojects

2020-06-05 Thread GitBox
elek merged pull request #949: URL: https://github.com/apache/hadoop-ozone/pull/949 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Created] (HDDS-3728) Support Storage space Level Quota for bucket

2020-06-05 Thread Simon Su (Jira)
Simon Su created HDDS-3728: -- Summary: Support Storage space Level Quota for bucket Key: HDDS-3728 URL: https://issues.apache.org/jira/browse/HDDS-3728 Project: Hadoop Distributed Data Store Issue Ty

[jira] [Created] (HDDS-3727) Support Storage space Level Quota for Volume

2020-06-05 Thread Simon Su (Jira)
Simon Su created HDDS-3727: -- Summary: Support Storage space Level Quota for Volume Key: HDDS-3727 URL: https://issues.apache.org/jira/browse/HDDS-3727 Project: Hadoop Distributed Data Store Issue Ty

[jira] [Created] (HDDS-3729) Support Namespace Level quota for Volume

2020-06-05 Thread Simon Su (Jira)
Simon Su created HDDS-3729: -- Summary: Support Namespace Level quota for Volume Key: HDDS-3729 URL: https://issues.apache.org/jira/browse/HDDS-3729 Project: Hadoop Distributed Data Store Issue Type:

[jira] [Created] (HDDS-3730) Support Namespace Level quota for bucket

2020-06-05 Thread Simon Su (Jira)
Simon Su created HDDS-3730: -- Summary: Support Namespace Level quota for bucket Key: HDDS-3730 URL: https://issues.apache.org/jira/browse/HDDS-3730 Project: Hadoop Distributed Data Store Issue Type:

[jira] [Updated] (HDDS-3703) Consider avoiding file lookup calls in writeChunk hotpath

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-3703: -- Fix Version/s: 0.6.0 Resolution: Fixed Status: Resolved (was: Patch Av

[GitHub] [hadoop-ozone] bshashikant merged pull request #1013: HDDS-3703. Avoid file lookup calls in writeChunk hotpath

2020-06-05 Thread GitBox
bshashikant merged pull request #1013: URL: https://github.com/apache/hadoop-ozone/pull/1013 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [hadoop-ozone] bshashikant commented on pull request #1013: HDDS-3703. Avoid file lookup calls in writeChunk hotpath

2020-06-05 Thread GitBox
bshashikant commented on pull request #1013: URL: https://github.com/apache/hadoop-ozone/pull/1013#issuecomment-639340839 Thanks @adoroszlai for the contribution. I have committed this. This is an automated message from the A

[jira] [Assigned] (HDDS-3725) Ozone shell client support setting quota

2020-06-05 Thread Simon Su (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Su reassigned HDDS-3725: -- Assignee: Simon Su > Ozone shell client support setting quota > ---

[jira] [Created] (HDDS-3731) Relative Unit Test

2020-06-05 Thread Simon Su (Jira)
Simon Su created HDDS-3731: -- Summary: Relative Unit Test Key: HDDS-3731 URL: https://issues.apache.org/jira/browse/HDDS-3731 Project: Hadoop Distributed Data Store Issue Type: Sub-task R

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #982: HDDS-3668. OzoneManager start fails with RocksDB error on downgrade to older version.

2020-06-05 Thread GitBox
adoroszlai commented on a change in pull request #982: URL: https://github.com/apache/hadoop-ozone/pull/982#discussion_r435773972 ## File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RDBStore.java ## @@ -149,6 +162,26 @@ public RDBStore(File dbFile

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1013: HDDS-3703. Avoid file lookup calls in writeChunk hotpath

2020-06-05 Thread GitBox
adoroszlai commented on pull request #1013: URL: https://github.com/apache/hadoop-ozone/pull/1013#issuecomment-639350440 Thanks @bshashikant for reviewing and merging this. This is an automated message from the Apache Git Ser

[jira] [Resolved] (HDDS-2108) TestContainerSmallFile#testReadWriteWithBCSId failure

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-2108. --- Fix Version/s: 0.6.0 Resolution: Cannot Reproduce > TestContainerSmallFile#test

[jira] [Resolved] (HDDS-3301) Freon is hanging if started when pipelines are not yet available

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-3301. --- Fix Version/s: 0.6.0 Resolution: Cannot Reproduce > Freon is hanging if started

[jira] [Updated] (HDDS-3732) Improve the performance of SCM by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Attachment: output.svg > Improve the performance of SCM by avoid UUID.toString > -

[jira] [Created] (HDDS-3732) Improve the performance of SCM by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3732: Summary: Improve the performance of SCM by avoid UUID.toString Key: HDDS-3732 URL: https://issues.apache.org/jira/browse/HDDS-3732 Project: Hadoop Distributed Data Store

[jira] [Updated] (HDDS-3732) Improve the performance of SCM by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Attachment: screenshot-1.png > Improve the performance of SCM by avoid UUID.toString > ---

[jira] [Updated] (HDDS-3732) Improve the performance of SCM by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Description: UUID.toString cost 7.23% CPU. We can store the string of UUID when init DatanodeDetails, and

[jira] [Created] (HDDS-3733) Improve the performance of SCM

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3733: Summary: Improve the performance of SCM Key: HDDS-3733 URL: https://issues.apache.org/jira/browse/HDDS-3733 Project: Hadoop Distributed Data Store Issue Type: Improv

[jira] [Updated] (HDDS-3732) Improve the performance of SCM by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Parent: HDDS-3733 Issue Type: Sub-task (was: Improvement) > Improve the performance of SCM by avo

[jira] [Resolved] (HDDS-1415) TestOzoneClientRetriesOnException.testGroupMismatchExceptionHandling is flaky

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-1415. --- Fix Version/s: 0.6.0 Resolution: Fixed > TestOzoneClientRetriesOnException.test

[jira] [Resolved] (HDDS-3298) TestFailureHandlingByClient.testDatanodeExclusionWithMajorityCommit is intermittent

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-3298. --- Fix Version/s: 0.6.0 Resolution: Fixed > TestFailureHandlingByClient.testDatano

[jira] [Created] (HDDS-3734) Improve the performance of SCM by improving queryNodeState

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3734: Summary: Improve the performance of SCM by improving queryNodeState Key: HDDS-3734 URL: https://issues.apache.org/jira/browse/HDDS-3734 Project: Hadoop Distributed Data Store

[jira] [Updated] (HDDS-3734) Improve the performance of SCM by improving queryNodeState

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Parent: HDDS-3733 Issue Type: Sub-task (was: Improvement) > Improve the performance of SCM by imp

[jira] [Resolved] (HDDS-2917) Add a different retry policy for watch requests

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-2917. --- Fix Version/s: 0.6.0 Resolution: Fixed > Add a different retry policy for watch

[jira] [Resolved] (HDDS-2152) Ozone client fails with OOM while writing a large (~300MB) key.

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-2152. --- Fix Version/s: 0.6.0 Resolution: Fixed > Ozone client fails with OOM while writ

[jira] [Updated] (HDDS-3733) Improve the performance of SCM

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3733: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[jira] [Updated] (HDDS-3733) Improve the performance of SCM

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3733: - Description: I start a ozone cluster with 1000 nodes, and run two weeks with heavy workload, and perf scm.

[jira] [Reopened] (HDDS-2152) Ozone client fails with OOM while writing a large (~300MB) key.

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee reopened HDDS-2152: --- > Ozone client fails with OOM while writing a large (~300MB) key. > --

[jira] [Resolved] (HDDS-2152) Ozone client fails with OOM while writing a large (~300MB) key.

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-2152. --- Resolution: Fixed > Ozone client fails with OOM while writing a large (~300MB) key. >

[jira] [Updated] (HDDS-3734) Improve the performance of SCM by improving queryNodeState

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Attachment: screenshot-1.png > Improve the performance of SCM by improving queryNodeState > --

[jira] [Commented] (HDDS-3083) AlreadyClosedException: SlidingWindow$Client:client-...->RAFT is closed

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126594#comment-17126594 ] Shashikant Banerjee commented on HDDS-3083: --- This should be addressed with https

[jira] [Updated] (HDDS-3734) Improve the performance of SCM by improving queryNodeState

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Description: queryNodeState cost 7.25% cpu. 1. we need not TreeSet.addAll which cost 3.86% cpu, because th

[jira] [Updated] (HDDS-3732) Improve the performance of SCM with 7.32% by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Summary: Improve the performance of SCM with 7.32% by avoid UUID.toString (was: Improve the performance o

[GitHub] [hadoop-ozone] runzhiwang opened a new pull request #1025: HDDS-3732. Improve the performance of SCM with 7.32% by avoid UUID.toString

2020-06-05 Thread GitBox
runzhiwang opened a new pull request #1025: URL: https://github.com/apache/hadoop-ozone/pull/1025 ## What changes were proposed in this pull request? avoid UUID.toString ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-3732

[jira] [Updated] (HDDS-3732) Improve the performance of SCM with 7.32% by avoid UUID.toString

2020-06-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3732: - Labels: pull-request-available (was: ) > Improve the performance of SCM with 7.32% by avoid UUID.

[GitHub] [hadoop-ozone] mukul1987 commented on pull request #911: HDDS-3564. Update Ozone to latest Ratis Snapshot (0.6.0-3596a58-SNAPSHOT).

2020-06-05 Thread GitBox
mukul1987 commented on pull request #911: URL: https://github.com/apache/hadoop-ozone/pull/911#issuecomment-639393533 Thanks @elek, Yes, I have updated the PR, this once again tries to update the Ratis version to get to a more stable version. -

[jira] [Updated] (HDDS-3734) Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Summary: Improve the performance of SCM with 3.86% by avoid TreeSet.addAll (was: Improve the performance

[jira] [Updated] (HDDS-3734) Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Description: TreeSet.addAll cost 7.25% cpu. we need not TreeSet.addAll which cost 3.86% cpu, because the

[jira] [Updated] (HDDS-3734) Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Description: TreeSet.addAll cost 3.86% cpu. we need not TreeSet.addAll which cost 3.86% cpu, because the

[jira] [Updated] (HDDS-3732) Improve the performance of SCM with 7.23% by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Summary: Improve the performance of SCM with 7.23% by avoid UUID.toString (was: Improve the performance o

[jira] [Updated] (HDDS-3732) Improve the performance of SCM with 7.23% by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Attachment: screenshot-2.png > Improve the performance of SCM with 7.23% by avoid UUID.toString >

[jira] [Updated] (HDDS-3732) Improve the performance of SCM with 7.23% by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[jira] [Updated] (HDDS-3732) Improve the performance of SCM with 7.23% by avoid UUID.toString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3732: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[GitHub] [hadoop-ozone] bshashikant opened a new pull request #1026: HDDS-3330. TestDeleteWithSlowFollower is still flaky.

2020-06-05 Thread GitBox
bshashikant opened a new pull request #1026: URL: https://github.com/apache/hadoop-ozone/pull/1026 ## What changes were proposed in this pull request? Increased the watch timeout so that key.close() succeeds on the same pipeline with majority commit. Added some precondition checks to

[jira] [Updated] (HDDS-3330) TestDeleteWithSlowFollower is still flaky

2020-06-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3330: - Labels: TriagePending ozone-flaky-test pull-request-available (was: TriagePending ozone-flaky-tes

[jira] [Resolved] (HDDS-1658) RaftRetryFailureException & AlreadyClosedException should not exclude pipeline from client

2020-06-05 Thread Shashikant Banerjee (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDDS-1658. --- Resolution: Not A Bug > RaftRetryFailureException & AlreadyClosedException should not

[GitHub] [hadoop-ozone] runzhiwang opened a new pull request #1027: HDDS-3734. Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread GitBox
runzhiwang opened a new pull request #1027: URL: https://github.com/apache/hadoop-ozone/pull/1027 ## What changes were proposed in this pull request? I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf scm. TreeSet.addAll cost 3.86% cpu

[jira] [Updated] (HDDS-3734) Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3734: - Labels: pull-request-available (was: ) > Improve the performance of SCM with 3.86% by avoid TreeS

[jira] [Created] (HDDS-3735) Improve the performance of SCM with 0.78 by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3735: Summary: Improve the performance of SCM with 0.78 by decrease the times of lock and unlock Key: HDDS-3735 URL: https://issues.apache.org/jira/browse/HDDS-3735 Project: Hadoo

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78 by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Parent: HDDS-3733 Issue Type: Sub-task (was: Improvement) > Improve the performance of SCM with 0

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78 by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Attachment: screenshot-1.png > Improve the performance of SCM with 0.78% by decrease the times of lock and

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Summary: Improve the performance of SCM with 0.78% by decrease the times of lock and unlock (was: Improv

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1014: HDDS-3710. Merge archived jacoco coverage results

2020-06-05 Thread GitBox
adoroszlai commented on pull request #1014: URL: https://github.com/apache/hadoop-ozone/pull/1014#issuecomment-639415271 @elek post-commit checks on master started failing with the error below since this PR was merged: ``` 2020-06-05T09:11:19.4479655Z [INFO] Loading execution data

[GitHub] [hadoop-ozone] runzhiwang opened a new pull request #1028: HDDS-3735. Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread GitBox
runzhiwang opened a new pull request #1028: URL: https://github.com/apache/hadoop-ozone/pull/1028 ## What changes were proposed in this pull request? I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf scm. lock and unlock in NodeStateM

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-3735: - Labels: pull-request-available (was: ) > Improve the performance of SCM with 0.78% by decrease th

[GitHub] [hadoop-ozone] nandakumar131 commented on pull request #1025: HDDS-3732. Improve the performance of SCM with 7.23% by avoid UUID.toString

2020-06-05 Thread GitBox
nandakumar131 commented on pull request #1025: URL: https://github.com/apache/hadoop-ozone/pull/1025#issuecomment-639423664 Thanks @runzhiwang for the improvement. We can also replace the `uuid.toString()` call inside `DatanodeDetails#toString()` to `strUuid`. -

[jira] [Updated] (HDDS-1648) Reduce layers of Ozone docker image

2020-06-05 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek updated HDDS-1648: -- Target Version/s: (was: 0.6.0) > Reduce layers of Ozone docker image > ---

[jira] [Updated] (HDDS-3290) Remove deprecated RandomKeyGenerator

2020-06-05 Thread Marton Elek (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Elek updated HDDS-3290: -- Target Version/s: 0.7.0 (was: 0.6.0) > Remove deprecated RandomKeyGenerator > -

[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1027: HDDS-3734. Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread GitBox
adoroszlai commented on a change in pull request #1027: URL: https://github.com/apache/hadoop-ozone/pull/1027#discussion_r435862235 ## File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java ## @@ -589,15 +589,15 @@ public

[jira] [Updated] (HDDS-3720) Datanode configuration object has wrong values

2020-06-05 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai updated HDDS-3720: --- Status: Patch Available (was: In Progress) > Datanode configuration object has wrong values >

[GitHub] [hadoop-ozone] lokeshj1703 commented on a change in pull request #1005: HDDS-3350. Ozone Retry Policy Improvements.

2020-06-05 Thread GitBox
lokeshj1703 commented on a change in pull request #1005: URL: https://github.com/apache/hadoop-ozone/pull/1005#discussion_r435870208 ## File path: pom.xml ## @@ -79,7 +79,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs ${ozone.version

[GitHub] [hadoop-ozone] lokeshj1703 commented on pull request #1005: HDDS-3350. Ozone Retry Policy Improvements.

2020-06-05 Thread GitBox
lokeshj1703 commented on pull request #1005: URL: https://github.com/apache/hadoop-ozone/pull/1005#issuecomment-639434963 /pending This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [hadoop-ozone] lokeshj1703 commented on a change in pull request #1005: HDDS-3350. Ozone Retry Policy Improvements.

2020-06-05 Thread GitBox
lokeshj1703 commented on a change in pull request #1005: URL: https://github.com/apache/hadoop-ozone/pull/1005#discussion_r435871997 ## File path: pom.xml ## @@ -79,7 +79,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs ${ozone.version

[jira] [Created] (HDDS-3736) Improve the performance of SCM with 1% by getContainerByBatch

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3736: Summary: Improve the performance of SCM with 1% by getContainerByBatch Key: HDDS-3736 URL: https://issues.apache.org/jira/browse/HDDS-3736 Project: Hadoop Distributed Data St

[jira] [Updated] (HDDS-3736) Improve the performance of SCM with 1% by getContainerByBatch

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3736: - Parent: HDDS-3733 Issue Type: Sub-task (was: Improvement) > Improve the performance of SCM with 1

[jira] [Updated] (HDDS-3736) Improve the performance of SCM with 1% by getContainerByBatch

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3736: - Attachment: screenshot-1.png > Improve the performance of SCM with 1% by getContainerByBatch > ---

[jira] [Updated] (HDDS-3736) Improve the performance of SCM with 1% by getContainerByBatch

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3736: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[GitHub] [hadoop-ozone] elek commented on pull request #1014: HDDS-3710. Merge archived jacoco coverage results

2020-06-05 Thread GitBox
elek commented on pull request #1014: URL: https://github.com/apache/hadoop-ozone/pull/1014#issuecomment-639444676 @adoroszlai Debugging right now. I think it's a conflict with HDDS-3627 / #975 Coverage report requires to unzip all the related classes. legacy/current jar files were exc

[jira] [Updated] (HDDS-3734) Improve the performance of SCM with 3.86% by avoid TreeSet.addAll

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3734: - Attachment: output.svg > Improve the performance of SCM with 3.86% by avoid TreeSet.addAll > -

[jira] [Updated] (HDDS-3736) Improve the performance of SCM with 1% by getContainerByBatch

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3736: - Attachment: output.svg > Improve the performance of SCM with 1% by getContainerByBatch > -

[jira] [Updated] (HDDS-3735) Improve the performance of SCM with 0.78% by decrease the times of lock and unlock

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3735: - Attachment: output.svg > Improve the performance of SCM with 0.78% by decrease the times of lock and > un

[jira] [Assigned] (HDDS-3719) Datanode may fail to stop

2020-06-05 Thread Attila Doroszlai (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Doroszlai reassigned HDDS-3719: -- Assignee: Attila Doroszlai > Datanode may fail to stop > - > >

[GitHub] [hadoop-ozone] runzhiwang commented on pull request #1025: HDDS-3732. Improve the performance of SCM with 7.23% by avoid UUID.toString

2020-06-05 Thread GitBox
runzhiwang commented on pull request #1025: URL: https://github.com/apache/hadoop-ozone/pull/1025#issuecomment-639464857 @nandakumar131 Thanks for review. I have updated. This is an automated message from the Apache Git Servi

[jira] [Created] (HDDS-3737) Improve the performance of OM

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3737: Summary: Improve the performance of OM Key: HDDS-3737 URL: https://issues.apache.org/jira/browse/HDDS-3737 Project: Hadoop Distributed Data Store Issue Type: Improve

[jira] [Created] (HDDS-3738) Improve the performance of OM with 13.47 by avoid UUID.fromString

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3738: Summary: Improve the performance of OM with 13.47 by avoid UUID.fromString Key: HDDS-3738 URL: https://issues.apache.org/jira/browse/HDDS-3738 Project: Hadoop Distributed Dat

[jira] [Updated] (HDDS-3738) Improve the performance of OM with 13.47 by avoid UUID.fromString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3738: - Parent: HDDS-3737 Issue Type: Sub-task (was: Improvement) > Improve the performance of OM with 13

[jira] [Updated] (HDDS-3738) Improve the performance of OM with 13.47 by avoid UUID.fromString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3738: - Attachment: screenshot-1.png > Improve the performance of OM with 13.47 by avoid UUID.fromString > ---

[jira] [Updated] (HDDS-3738) Improve the performance of OM with 13.47 by avoid UUID.fromString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3738: - Description: I start a ozone cluster with 1000 datanodes, and run two weeks with heavy workload, and perf

[jira] [Resolved] (HDDS-3621) Separate client/server/admin proto files of HDDS to separated subprojects

2020-06-05 Thread Nanda kumar (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar resolved HDDS-3621. --- Fix Version/s: 0.6.0 Resolution: Fixed > Separate client/server/admin proto files of HDDS to se

[GitHub] [hadoop-ozone] captainzmc commented on pull request #814: HDDS-3286. BasicOzoneFileSystem support batchDelete.

2020-06-05 Thread GitBox
captainzmc commented on pull request #814: URL: https://github.com/apache/hadoop-ozone/pull/814#issuecomment-639486496 Had rebased master and fixed conflict. PR can continue to be reviewed. cc @timmylicheng @ChenSammi @xiaoyuyao Thanks.

[jira] [Updated] (HDDS-3738) Improve the performance of OM with 13.47 by avoid UUID.fromString

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3738: - Attachment: output.svg > Improve the performance of OM with 13.47 by avoid UUID.fromString > -

[jira] [Created] (HDDS-3739) Improve datanode performance by avoid get waitting deleted blocks from RocksDB

2020-06-05 Thread runzhiwang (Jira)
runzhiwang created HDDS-3739: Summary: Improve datanode performance by avoid get waitting deleted blocks from RocksDB Key: HDDS-3739 URL: https://issues.apache.org/jira/browse/HDDS-3739 Project: Hadoop Di

[jira] [Updated] (HDDS-3739) Improve datanode performance by avoid get waitting deleted blocks from RocksDB

2020-06-05 Thread runzhiwang (Jira)
[ https://issues.apache.org/jira/browse/HDDS-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] runzhiwang updated HDDS-3739: - Attachment: screenshot-1.png > Improve datanode performance by avoid get waitting deleted blocks from Rock

  1   2   >