[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815682#comment-16815682 ] Thiruvel Thirumoolan commented on HBASE-20546: -- Thanks Andy, I think I had a draft patch of it a while back, shelved. Will resume work on it. > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly created for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815606#comment-16815606 ] Andrew Purtell commented on HBASE-20546: Reviewing old issues looks like I dropped the ball on feedback. Yes that would be ok, if this work gets picked up again at some point. As for now I'm going to unschedule it. > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement > Components: Balancer >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly created for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505127#comment-16505127 ] Thiruvel Thirumoolan commented on HBASE-20546: -- I can move this map to ServerManager, so its always updated. Would that be ok? Since its called from picker, its significant. It came up when I profiled balancer on our cluster setup. > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.6 > > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly created for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471212#comment-16471212 ] Andrew Purtell commented on HBASE-20546: I think we may need to make hostToServerName a concurrent hash map or otherwise protect it with synchronized access. Otherwise the load balancer could crash with CME. Map needs to be cleared first too, like [~chia7712] mentioned. How much are we saving? Seems the rebuild of the map is still pretty expensive. Can we incrementally update this state somehow instead? Or reuse state from somewhere else? Is there another place in the master we do frequent lookups of ServerNames from hostname? In the AM somewhere maybe? > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly created for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471173#comment-16471173 ] Thiruvel Thirumoolan commented on HBASE-20546: -- Thanks [~chia7712]. Good point. It looks like ClusterStatusChore will be updating clusterStatus. Its intention is to make StochasticBalancer better by updating regionload. Not sure why regionFinder's clusterstatus needs to be updated. I think we can have a setClusterStatus and updateClusterStatus API, so ClusterStatusChore can use the latter. I am not sure if updating clusterstatus or re-initializing the hostserver map I introduce in regionfinder in the middle of balance is a good idea. What do you think? > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly created for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[ https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470175#comment-16470175 ] Chia-Ping Tsai commented on HBASE-20546: The #setClusterStatus may be called many times. Should we clear the hostToServerName when calling #setClusterStatus? Also, Is there concurrent issue in changing hostToServerName? > Improve perf of RegionLocationFinder.mapHostNameToServerName > > > Key: HBASE-20546 > URL: https://issues.apache.org/jira/browse/HBASE-20546 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.4.4, 2.0.0 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 1.5.0, 2.0.1, 1.4.5 > > Attachments: HBASE-20546.branch-1.4.001.patch > > > RegionLocationFinder.getTopBlockLocations() is called multiple times during > balancer. While profiling on a large table balance, mapHostNameToServerName() > seem to take a lot of time. One of the maps is repeatedly consumed for each > iteration, while we can just initialize it once. > Goes into both branch-1 and branch-2, although patches differ slightly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20546) Improve perf of RegionLocationFinder.mapHostNameToServerName
[
https://issues.apache.org/jira/browse/HBASE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468143#comment-16468143
]
Hadoop QA commented on HBASE-20546:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} branch-1.4 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
25s{color} | {color:red} hbase-server in branch-1.4 failed with JDK v1.8.0_172.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hbase-server in branch-1.4 failed with JDK v1.7.0_181.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
40s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_172.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_172. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
16s{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_181.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_181. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 58s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 37s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:74e3133 |
| JIRA Issue | HBASE-20546 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922537/HBASE-20546.branch-1.4.001.patch
|
| Optional Tests | asflicense javac javadoc
