[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981485#comment-13981485 ] Hudson commented on HBASE-10873: FAILURE: Integrated in HBase-TRUNK #5117 (See [https://builds.apache.org/job/HBase-TRUNK/5117/]) HBASE-10873 Control number of regions assigned to backup masters (jxiang: rev 1590078) * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/ClusterLoadState.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/FavoredNodeLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/SimpleLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancer.java Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch, hbase-10873_v3.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[
https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13978542#comment-13978542
]
Hadoop QA commented on HBASE-10873:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641492/hbase-10873_v3.patch
against trunk revision .
ATTACHMENT ID: 12641492
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9372//console
This message is automatically generated.
Control number of regions assigned to backup masters
Key: HBASE-10873
URL: https://issues.apache.org/jira/browse/HBASE-10873
Project: HBase
Issue Type: Improvement
Components: Balancer
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10873.patch, hbase-10873_v2.patch,
hbase-10873_v3.patch
By default, a backup master is treated just like another regionserver. So it
can host as many regions as other regionserver does. When the backup master
becomes the active one, region balancer needs to move those user regions on
this master to other region servers. To minimize the impact, it's better not
to assign too many regions on backup masters. It may not be good to leave the
backup masters idle and not host any region either.
We should make this adjustable so that users can control how many regions to
assign to each backup master.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979125#comment-13979125 ] Jimmy Xiang commented on HBASE-10873: - The test failure is not related. If no objection, I will commit v3 tomorrow. We can change the default setting in a separate issue if needed. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch, hbase-10873_v3.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977144#comment-13977144 ] Jimmy Xiang commented on HBASE-10873: - Can we get this in and deal with the default values in a followup issue? Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977199#comment-13977199 ] stack commented on HBASE-10873: --- Good by me. Thinking on it, I think it'd be fine to flag in a release note that Master and BUMaster will carry full-complement of regions after upgrade (unless you configure it to work the old way). Having this option where *Master's carry a lighter loading would be a nice-to-have though I think it will be little used. [~jxiang] If we set it so backup masters should NOT carry regions, does it use this code path -- the one in this patch? See what [~enis] says. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977232#comment-13977232 ] Jimmy Xiang commented on HBASE-10873: - bq. If we set it so backup masters should NOT carry regions, does it use this code path – the one in this patch? No, it doesn't use this patch. It is a different configuration. Probably it's better to do some consolidation and use just the configuration in this patch? Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13977537#comment-13977537 ] stack commented on HBASE-10873: --- bq. Probably it's better to do some consolidation and use just the configuration in this patch? It would make this patch indispensible (smile). Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13975844#comment-13975844 ] Enis Soztutar commented on HBASE-10873: --- bq. Should we set the default value to 1 (backup master is treated the same as a normal region server) or a big number (less regions)? If big, how big should be proper? I was thinking of setting the value to 0, which gives us the 0.98- behavior of backup masters not hosting any regions. If users want to use extra capacity they can enable this setting manually. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13975883#comment-13975883 ] Jimmy Xiang commented on HBASE-10873: - The patch is just for trunk. I thought we prefer 1.0 to have the new behavior. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13975974#comment-13975974 ] stack commented on HBASE-10873: --- bq. I was thinking of setting the value to 0, which gives us the 0.98- behavior of backup masters not hosting any regions. If users want to use extra capacity they can enable this setting manually. I suggest we let this issue in and then elsewhere discuss what 1.0 rolls out with elsewhere -- in subtask on the 1.0 issue. I'd be in favor of master and backup masters carrying full complement of regions in 1.0 (if we can ensure that the master handlers are processed ahead of any others); i.e. more radical than Jimmy has it his adding support for master and backup masters carrying 'light' loadings. Sounds like you'd like us to replicate the old deploy layout [~enis] but with option to move to the 'new'. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976006#comment-13976006 ] Enis Soztutar commented on HBASE-10873: --- bq. Sounds like you'd like us to replicate the old deploy layout Enis Soztutar but with option to move to the 'new'. Yes. Least amount of surprises. Since we are right now defaulting to master having no user level regions, we should also default to backup masters not having any user level regions. It may be ok to change the defaults so that both active and backup masters have normal region load (weight 1), but I think we should be consistent for region loads of active and backup masters. Since META is already shared in 0.98- setups, it should be fine to have master be just another region server. In that case, we do not even need this issue? Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976031#comment-13976031 ] stack commented on HBASE-10873: --- bq. In that case, we do not even need this issue? Perhaps. I'm not sure how we currently keep backup masters 'clear'. Wouldn't we want it done in the LB? Or how is it done currently [~jxiang]? Thanks. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976050#comment-13976050 ] Jimmy Xiang commented on HBASE-10873: - bq. In that case, we do not even need this issue? This patch makes it possible to put some load on backup masters. So that users can make adjustments per their need. I hope we don't need this issue, if not for the region moving concern during master failover. bq. I'm not sure how we currently keep backup masters 'clear'. Wouldn't we want it done in the LB? Yes, it's done in the LB. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13974271#comment-13974271 ] Enis Soztutar commented on HBASE-10873: --- bq. I think that's the issue we'd like to enhance. We want the region server hosting meta doesn't host too many heavy regions so that we can improve the meta availability, right? We already have separate priority handlers and separate wal, etc in place right? Plus the active master right now only hosts system tables, so this is not a problem for backup masters right? Are you concerned with the failover? bq. Currently, we already support all or nothing. With this patch, it will make it more flexible so that users don't have to leave those backup masters totally idle, they don't need to worry too much about region movement/data locality in case each region server hosts many regions. Fair enough, but configuring this for a cluster is still not clear. What value should we default? What value should a 20 node cluster use, etc. It might be ok if we default to backups not hosting any regions. Then if the user prefers to use the capacity, they can explicitly request so. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13974436#comment-13974436 ] Jimmy Xiang commented on HBASE-10873: - bq Are you concerned with the failover? Yes, that's the concern here. I was thinking later on, we can open the meta on the backup master as well after we support opening a region in multiple servers. This way, it will be much fast to fail over an active master change. As to the weight, basically, it means how many regions to host by a backup master compared to a normal region server. The default is 4, i.e. by default, the number of regions opened on a backup master is 1/4 of the number of regions opened on a normal region server. Should we set the default value to 1 (backup master is treated the same as a normal region server) or a big number (less regions)? If big, how big should be proper? Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13973598#comment-13973598 ] Jimmy Xiang commented on HBASE-10873: - [~enis], what do you think? Are you ok to get this patch in? Thanks. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[
https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970214#comment-13970214
]
Hadoop QA commented on HBASE-10873:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12640327/hbase-10873_v2.patch
against trunk revision .
ATTACHMENT ID: 12640327
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9297//console
This message is automatically generated.
Control number of regions assigned to backup masters
Key: HBASE-10873
URL: https://issues.apache.org/jira/browse/HBASE-10873
Project: HBase
Issue Type: Improvement
Components: Balancer
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10873.patch, hbase-10873_v2.patch
By default, a backup master is treated just like another regionserver. So it
can host as many regions as other regionserver does. When the backup master
becomes the active one, region balancer needs to move those user regions on
this master to other region servers. To minimize the impact, it's better not
to assign too many regions on backup masters. It may not be good to leave the
backup masters idle and not host any region either.
We should make this adjustable so that users can control how many regions to
assign to each backup master.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970226#comment-13970226 ] Matteo Bertozzi commented on HBASE-10873: - +1, v2 looks good to me Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970345#comment-13970345 ] Enis Soztutar commented on HBASE-10873: --- Is this a good idea? Today META/ns tables are already shared with the other tables, and there is little special treatment for the region server hosting meta. In 0.98- deployments, in deployments where master and backup masters are co-located with region servers, this would basically reduce the capacity of the cluster. Otherwise in deployments where backup masters and active master is not co-located, the users will probably want backup masters to not host any region server (since it was the case already). I think we should do a all-or-nothing approach rather that this weight based config which is just yet another parameter to worry about. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[ https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970373#comment-13970373 ] Jimmy Xiang commented on HBASE-10873: - bq. Today META/ns tables are already shared with the other tables, and there is little special treatment for the region server hosting meta. I think that's the issue we'd like to enhance. We want the region server hosting meta doesn't host too many heavy regions so that we can improve the meta availability, right? bq. I think we should do a all-or-nothing approach rather that this weight based config which is just yet another parameter to worry about. Currently, we already support all or nothing. With this patch, it will make it more flexible so that users don't have to leave those backup masters totally idle, they don't need to worry too much about region movement/data locality in case each region server hosts many regions. Control number of regions assigned to backup masters Key: HBASE-10873 URL: https://issues.apache.org/jira/browse/HBASE-10873 Project: HBase Issue Type: Improvement Components: Balancer Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: hbase-10873.patch, hbase-10873_v2.patch By default, a backup master is treated just like another regionserver. So it can host as many regions as other regionserver does. When the backup master becomes the active one, region balancer needs to move those user regions on this master to other region servers. To minimize the impact, it's better not to assign too many regions on backup masters. It may not be good to leave the backup masters idle and not host any region either. We should make this adjustable so that users can control how many regions to assign to each backup master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10873) Control number of regions assigned to backup masters
[
https://issues.apache.org/jira/browse/HBASE-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962128#comment-13962128
]
Hadoop QA commented on HBASE-10873:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639016/hbase-10873.patch
against trunk revision .
ATTACHMENT ID: 12639016
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.procedure.TestZKProcedure
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9216//console
This message is automatically generated.
Control number of regions assigned to backup masters
Key: HBASE-10873
URL: https://issues.apache.org/jira/browse/HBASE-10873
Project: HBase
Issue Type: Improvement
Components: Balancer
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.99.0
Attachments: hbase-10873.patch
By default, a backup master is treated just like another regionserver. So it
can host as many regions as other regionserver does. When the backup master
becomes the active one, region balancer needs to move those user regions on
this master to other region servers. To minimize the impact, it's better not
to assign too many regions on backup masters. It may not be good to leave the
backup masters idle and not host any region either.
We should make this adjustable so that users can control how many regions to
assign to each backup master.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
