[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482964#comment-16482964 ] Thiruvel Thirumoolan commented on HBASE-20545: -- Thanks [~apurtell], [~yuzhih...@gmail.com] and [~aoxiang] for reviews. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477737#comment-16477737
]
Hudson commented on HBASE-20545:
Results for branch branch-1.4
[build #322 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/322/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/322//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/322//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/322//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve performance of BaseLoadBalancer.retainAssignment
>
>
> Key: HBASE-20545
> URL: https://issues.apache.org/jira/browse/HBASE-20545
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 1.4.4, 2.0.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.5
>
> Attachments: HBASE-20545.branch-1.4.001.patch,
> HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch
>
>
> I was measuring perf at scale with a 1m region table and noticed some
> improvements can be made to BaseLoadBalancer.retainAssignment().
> retainAssignment() spends a few mins to enable a 1m regions table and also
> generates a lot of objects unnecessarily. This jira is to make the most
> common case go faster with very minimal changes. A slightly modified version
> of this patch takes about 5 seconds for a 1m region table ignoring the time
> spent in createCluster(). I think locality can be refreshed during master
> startup in different ways without taking time in retainAssignment, but will
> follow up on that in subsequent jiras. Leaving it untouched here, but wanted
> to call out the time taken without that method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477630#comment-16477630
]
Hudson commented on HBASE-20545:
Results for branch branch-1.3
[build #329 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/329/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/329//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/329//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/329//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve performance of BaseLoadBalancer.retainAssignment
>
>
> Key: HBASE-20545
> URL: https://issues.apache.org/jira/browse/HBASE-20545
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 1.4.4, 2.0.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.5
>
> Attachments: HBASE-20545.branch-1.4.001.patch,
> HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch
>
>
> I was measuring perf at scale with a 1m region table and noticed some
> improvements can be made to BaseLoadBalancer.retainAssignment().
> retainAssignment() spends a few mins to enable a 1m regions table and also
> generates a lot of objects unnecessarily. This jira is to make the most
> common case go faster with very minimal changes. A slightly modified version
> of this patch takes about 5 seconds for a 1m region table ignoring the time
> spent in createCluster(). I think locality can be refreshed during master
> startup in different ways without taking time in retainAssignment, but will
> follow up on that in subsequent jiras. Leaving it untouched here, but wanted
> to call out the time taken without that method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477224#comment-16477224
]
Hudson commented on HBASE-20545:
Results for branch branch-1
[build #316 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/316/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/316//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/316//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/316//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Improve performance of BaseLoadBalancer.retainAssignment
>
>
> Key: HBASE-20545
> URL: https://issues.apache.org/jira/browse/HBASE-20545
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 1.4.4, 2.0.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.5
>
> Attachments: HBASE-20545.branch-1.4.001.patch,
> HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch
>
>
> I was measuring perf at scale with a 1m region table and noticed some
> improvements can be made to BaseLoadBalancer.retainAssignment().
> retainAssignment() spends a few mins to enable a 1m regions table and also
> generates a lot of objects unnecessarily. This jira is to make the most
> common case go faster with very minimal changes. A slightly modified version
> of this patch takes about 5 seconds for a 1m region table ignoring the time
> spent in createCluster(). I think locality can be refreshed during master
> startup in different ways without taking time in retainAssignment, but will
> follow up on that in subsequent jiras. Leaving it untouched here, but wanted
> to call out the time taken without that method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476689#comment-16476689 ] Andrew Purtell commented on HBASE-20545: Hmm, this was committed to master and branch-2 on May 9. Why is this issue still open? Please resolve issues when commits take place. Please commit to all branches at one time, or file a backport issue for separate commit to e.g. branch-1. I see this not committed to branch-1, branch-1.4, branch-1.3, or branch-2.0. I am not going to touch branch-2.0 per [~stack]'s request. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16476572#comment-16476572 ] Andrew Purtell commented on HBASE-20545: Going to commit this shortly, then > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475014#comment-16475014 ] Andrew Purtell commented on HBASE-20545: That test failure doesn't look related. +1 > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472881#comment-16472881
]
Hadoop QA commented on HBASE-20545:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} branch-1.4 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
20s{color} | {color:red} hbase-server in branch-1.4 failed with JDK v1.8.0_172.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} | {color:red} hbase-server in branch-1.4 failed with JDK v1.7.0_181.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
20s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
12s{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_172.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 12s{color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_172. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
15s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_181.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 15s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_181. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} hbase-server: The patch generated 0 new + 77
unchanged - 1 fixed = 77 total (was 78) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 13s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:74e3133 |
| JIRA Issue |
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472831#comment-16472831 ] Thiruvel Thirumoolan commented on HBASE-20545: -- Doesn't look the failure is related to my patch, have resubmitted 002 patch to trigger the build again. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-1.4.002.patch, HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16472065#comment-16472065 ] Ted Yu commented on HBASE-20545: Please check branch-1 QA test failures. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471272#comment-16471272
]
Hudson commented on HBASE-20545:
Results for branch branch-2
[build #719 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/719/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/719//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/719//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/719//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve performance of BaseLoadBalancer.retainAssignment
>
>
> Key: HBASE-20545
> URL: https://issues.apache.org/jira/browse/HBASE-20545
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 1.4.4, 2.0.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.4.5
>
> Attachments: HBASE-20545.branch-1.4.001.patch,
> HBASE-20545.branch-2.001.patch
>
>
> I was measuring perf at scale with a 1m region table and noticed some
> improvements can be made to BaseLoadBalancer.retainAssignment().
> retainAssignment() spends a few mins to enable a 1m regions table and also
> generates a lot of objects unnecessarily. This jira is to make the most
> common case go faster with very minimal changes. A slightly modified version
> of this patch takes about 5 seconds for a 1m region table ignoring the time
> spent in createCluster(). I think locality can be refreshed during master
> startup in different ways without taking time in retainAssignment, but will
> follow up on that in subsequent jiras. Leaving it untouched here, but wanted
> to call out the time taken without that method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470829#comment-16470829 ] Thiruvel Thirumoolan commented on HBASE-20545: -- [~yuzhih...@gmail.com], Looks like the build you triggered passed. Since existing tests were sufficient, I didn't add any. Can we also get this into branch-2? > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 3.0.0, 1.5.0, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470328#comment-16470328
]
Hudson commented on HBASE-20545:
Results for branch master
[build #326 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/326/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/326//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/326//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/326//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Improve performance of BaseLoadBalancer.retainAssignment
>
>
> Key: HBASE-20545
> URL: https://issues.apache.org/jira/browse/HBASE-20545
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 1.4.4, 2.0.0
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Major
> Fix For: 1.5.0, 2.0.1, 1.4.5
>
> Attachments: HBASE-20545.branch-1.4.001.patch,
> HBASE-20545.branch-2.001.patch
>
>
> I was measuring perf at scale with a 1m region table and noticed some
> improvements can be made to BaseLoadBalancer.retainAssignment().
> retainAssignment() spends a few mins to enable a 1m regions table and also
> generates a lot of objects unnecessarily. This jira is to make the most
> common case go faster with very minimal changes. A slightly modified version
> of this patch takes about 5 seconds for a 1m region table ignoring the time
> spent in createCluster(). I think locality can be refreshed during master
> startup in different ways without taking time in retainAssignment, but will
> follow up on that in subsequent jiras. Leaving it untouched here, but wanted
> to call out the time taken without that method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469851#comment-16469851
]
Hadoop QA commented on HBASE-20545:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} hbase-server: The patch generated 0 new + 60
unchanged - 1 fixed = 60 total (was 61) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 19s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}128m
36s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-20545 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922694/HBASE-20545.branch-2.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-139-generic #188-Ubuntu SMP Tue
Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / cba8d2f |
| maven | version: Apache Maven 3.0.5
(r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 13:51:28+) |
| Default Java | 1.8.0_172 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12775/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12775/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Improve performance of
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469711#comment-16469711 ] Ted Yu commented on HBASE-20545: Triggered https://builds.apache.org/job/PreCommit-HBASE-Build/12775/ > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469710#comment-16469710 ] Ted Yu commented on HBASE-20545: >From https://builds.apache.org/job/PreCommit-HBASE-Build/12771/console : 19:03:47 # There is insufficient memory for the Java Runtime Environment to continue. 19:03:47 # Cannot create GC thread. Out of system resources. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469292#comment-16469292 ] Thiruvel Thirumoolan commented on HBASE-20545: -- Uploaded patch for master branch. The only change from branch-1 patch is RegionInfo instead of HRegionInfo. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch, > HBASE-20545.branch-2.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469222#comment-16469222 ] Thiruvel Thirumoolan commented on HBASE-20545: -- Linking HBASE-20548 for improvements to locality refresh during master startup. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468876#comment-16468876 ] Ted Yu commented on HBASE-20545: Please attach patch for master branch. Thanks > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468871#comment-16468871 ] binlijin commented on HBASE-20545: -- +1, LGTM > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468204#comment-16468204 ] Ted Yu commented on HBASE-20545: [~aoxiang]: Can you take a look at the patch ? > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[
https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468145#comment-16468145
]
Hadoop QA commented on HBASE-20545:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} branch-1.4 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
24s{color} | {color:red} hbase-server in branch-1.4 failed with JDK v1.8.0_172.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hbase-server in branch-1.4 failed with JDK v1.7.0_181.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_172.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_172. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_181.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_181. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} hbase-server: The patch generated 0 new + 77
unchanged - 1 fixed = 77 total (was 78) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 0s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 6s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:74e3133 |
| JIRA Issue | HBASE-20545 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-20545) Improve performance of BaseLoadBalancer.retainAssignment
[ https://issues.apache.org/jira/browse/HBASE-20545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467984#comment-16467984 ] Thiruvel Thirumoolan commented on HBASE-20545: -- Uploaded patch for 1.4, will also upload one for branch 2.x. The approach is that - for most scenarios, the servers are the same when retainAssignment is called. But we unnecessarily create Cluster object and populate it for every region. At scale, this times time (array copies in Cluster.doAssignRegion) and also lot of garbage. Since thats not common case, I moved out that to a separate loop and only initialize them when random assignment is required. So in the worst case, we still take that time, but for common scenarios, its much faster and less memory garbage. We can make the worst case scenario also faster, but thats later. > Improve performance of BaseLoadBalancer.retainAssignment > > > Key: HBASE-20545 > URL: https://issues.apache.org/jira/browse/HBASE-20545 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20545.branch-1.4.001.patch > > > I was measuring perf at scale with a 1m region table and noticed some > improvements can be made to BaseLoadBalancer.retainAssignment(). > retainAssignment() spends a few mins to enable a 1m regions table and also > generates a lot of objects unnecessarily. This jira is to make the most > common case go faster with very minimal changes. A slightly modified version > of this patch takes about 5 seconds for a 1m region table ignoring the time > spent in createCluster(). I think locality can be refreshed during master > startup in different ways without taking time in retainAssignment, but will > follow up on that in subsequent jiras. Leaving it untouched here, but wanted > to call out the time taken without that method. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
