[jira] [Commented] (HBASE-22618) added the possibility to load custom cost functions
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078963#comment-17078963
]
Pierre Zemb commented on HBASE-22618:
-
This is pretty solid release note Nick, thanks a lot!
> added the possibility to load custom cost functions
> ---
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementati
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077965#comment-17077965
]
Pierre Zemb commented on HBASE-22618:
-
We published a blogpost about that contribution on my company's blog:
[https://www.ovh.com/blog/contributing-to-apache-hbase-custom-data-balancing/]
In it, you will find our context, why this patch was needed, and an example
with our custom balance function.
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991772#comment-16991772 ] Pierre Zemb commented on HBASE-23073: - Thank you [~wchevreuil] for your patience ! > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0, 1.6.0 > > Attachments: HBASE-23073.branch-1.0002.patch, > HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986769#comment-16986769 ] Pierre Zemb commented on HBASE-23073: - https://github.com/apache/hbase/pull/894 > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.0002.patch, > HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986759#comment-16986759 ] Pierre Zemb commented on HBASE-23073: - Directly on github? Doing :) > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.0002.patch, > HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963992#comment-16963992 ] Pierre Zemb commented on HBASE-23073: - I added a patch for branch-1 [~wchevreuil] > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Zemb updated HBASE-23073: Attachment: HBASE-23073.branch-1.001.patch > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-23073.branch-1.001.patch > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958707#comment-16958707 ] Pierre Zemb commented on HBASE-23073: - I will!, thanks again [~wchevreuil] for helping me contribute to HBase :) > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: New Feature > Components: master >Affects Versions: 3.0.0 >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941255#comment-16941255
]
Pierre Zemb edited comment on HBASE-22618 at 9/30/19 6:57 PM:
--
I created the associated JIRA here:
https://issues.apache.org/jira/browse/HBASE-23073
cc [~wchevreuil]
was (Author: pierrez):
I created the associated JIRA here:
https://issues.apache.org/jira/browse/HBASE-23073
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941255#comment-16941255
]
Pierre Zemb commented on HBASE-22618:
-
I created the associated JIRA here:
https://issues.apache.org/jira/browse/HBASE-23073
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
>
[jira] [Commented] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
[ https://issues.apache.org/jira/browse/HBASE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941253#comment-16941253 ] Pierre Zemb commented on HBASE-23073: - I just opened the PR: https://github.com/apache/hbase/pull/677 > Add an optional costFunction to balance regions according to a capacity rule > > > Key: HBASE-23073 > URL: https://issues.apache.org/jira/browse/HBASE-23073 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Pierre Zemb >Assignee: Pierre Zemb >Priority: Minor > > Based on the work in > [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can > now load custom costFunctions inside the main balancer used by HBase. As an > example, we like like to add upstream an optional cost function called > HeterogeneousRegionCountCostFunction that will deal with our issue: how to > balance regions according to the capacity of a RS instead of using the > RegionCountSkewCostFunction that is trying to avoid skew. > A rule file is loaded from HDFS before balancing. It contains lines of rules. > A rule is composed of a regexp for hostname, and a limit. For example, we > could have: > * rs[0-9] 200 > * rs1[0-9] 50 > RegionServers with hostname matching the first rules will have a limit of > 200, and the others 50. If there's no match, a default is set. > Thanks to the rule, we have two informations: the max number of regions for > this cluster, and the rules for each servers. HeterogeneousBalancer will try > to balance regions according to their capacity. > Let's take an example. Let's say that we have 20 RS: > 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can > handle 200 regions. > 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each > can support 50 regions. > Based on the following rules: > rs[0-9] 200 > rs1[0-9] 50 > The second group is overloaded, whereas the first group has plenty of space. > Moving a region from the first group to the second should provide a lower > cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-23073) Add an optional costFunction to balance regions according to a capacity rule
Pierre Zemb created HBASE-23073: --- Summary: Add an optional costFunction to balance regions according to a capacity rule Key: HBASE-23073 URL: https://issues.apache.org/jira/browse/HBASE-23073 Project: HBase Issue Type: Improvement Components: master Reporter: Pierre Zemb Assignee: Pierre Zemb Based on the work in [HBASE-22618|https://issues.apache.org/jira/browse/HBASE-22618], users can now load custom costFunctions inside the main balancer used by HBase. As an example, we like like to add upstream an optional cost function called HeterogeneousRegionCountCostFunction that will deal with our issue: how to balance regions according to the capacity of a RS instead of using the RegionCountSkewCostFunction that is trying to avoid skew. A rule file is loaded from HDFS before balancing. It contains lines of rules. A rule is composed of a regexp for hostname, and a limit. For example, we could have: * rs[0-9] 200 * rs1[0-9] 50 RegionServers with hostname matching the first rules will have a limit of 200, and the others 50. If there's no match, a default is set. Thanks to the rule, we have two informations: the max number of regions for this cluster, and the rules for each servers. HeterogeneousBalancer will try to balance regions according to their capacity. Let's take an example. Let's say that we have 20 RS: 10 RS, named through rs0 to rs9 loaded with 60 regions each, and each can handle 200 regions. 10 RS, named through rs10 to rs19 loaded with 60 regions each, and each can support 50 regions. Based on the following rules: rs[0-9] 200 rs1[0-9] 50 The second group is overloaded, whereas the first group has plenty of space. Moving a region from the first group to the second should provide a lower cost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925235#comment-16925235
]
Pierre Zemb commented on HBASE-22618:
-
Yes I can do that! How can I provide you some Release Notes?
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16917988#comment-16917988
]
Pierre Zemb commented on HBASE-22618:
-
Hi [~wchevreuil] ! Thanks for the help getting my patch into the different
branches! I do have some questions:
* 1.5 is the only target for 1.X? Can we target 1.4?
* The second patch is missing, making the Jira not resolved. First patch was
about opening the possibility to add cost functions, but we want to also
provide an example of a cost function, by adding one that is allowing for
Heterogeneous cluster, as stated by the ticket. Should I create another Jira?
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-1.002.patch, HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> acc
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-2.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggab
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: (was: HBASE-22618.branch-2.001.patch)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: (was: HBASE-22618.branch-1.001.patch)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-1.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggab
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-1.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggab
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: (was: HBASE-22618.branch-1.001.patch)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions ins
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-1.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggab
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: (was: HBASE-22618.branch-1.001.patch)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions ins
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-2.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions instead of thi
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: (was: HBASE-22618.branch-2.001.patch)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions ins
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913395#comment-16913395
]
Pierre Zemb commented on HBASE-22618:
-
Yeah sorry, I need to have a look at this. I will try to fix it ASAP :)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quot
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910038#comment-16910038
]
Pierre Zemb commented on HBASE-22618:
-
Done!
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balance
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-1.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggab
[jira] [Comment Edited] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910027#comment-16910027
]
Pierre Zemb edited comment on HBASE-22618 at 8/18/19 6:51 PM:
--
Wow, thanks for the merge [~wchevreuil]!
Should I be working on the branch-1 as well? It is my main target after all.
I dit generated some patches, are they ok for you? Not quite used to patches :)
was (Author: pierrez):
Wow, thanks for the merge [~wchevreuil]!
Should I be working on the branch-1 as well? It is my main target after all.
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-1.001.patch,
> HBASE-22618.branch-2.001.patch, HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Attachment: HBASE-22618.branch-2.001.patch
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.branch-2.001.patch,
> HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions instead of thi
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910027#comment-16910027
]
Pierre Zemb commented on HBASE-22618:
-
Wow, thanks for the merge [~wchevreuil]!
Should I be working on the branch-1 as well? It is my main target after all.
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.2.1, 2.1.6, 1.4.11, 2.1.7
>Reporter: Pierre Zemb
>Assignee: Pierre Zemb
>Priority: Major
> Attachments: HBASE-22618.master.001.patch
>
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898356#comment-16898356
]
Pierre Zemb commented on HBASE-22618:
-
I just opened the PR [https://github.com/apache/hbase/pull/439]
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions instead of this static list of cost
> functions|[https://github.com/apache/hbase/blob/baf3
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16894997#comment-16894997
]
Pierre Zemb commented on HBASE-22618:
-
Hi!
I'm back from my holidays, I will rework it using a Map, this is will simplify
the implementation indeed, thanks for the suggestion. And I will create the PR
:)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable co
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883853#comment-16883853
]
Pierre Zemb commented on HBASE-22618:
-
Hi!
I worked a bit on the PR. I created branch from master and started it again.
This [new
branch|https://github.com/PierreZ/hbase/commit/6b2edab4e5059913090e7b0db2e90d67afb15729]
is based on master this time. I decided to go through the first technical
point, which is how we are exposing custom cost functions.
This first implementation is only capable of adding new cost functions. I tried
to load them in a dynamic manner, but I don't like how the code turned-up: as
some functions are needed elsewhere, there is an ugly switch case matching for
already loaded functions, visible here:
[https://gist.github.com/PierreZ/1dc5ba6e9217cb97308cbe25d5b3#file-testfullyconfigurable-java-L212]
I decided to make only extension available, as every cost function implements a
multiplier that can disabled it. What do you think about this?
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880361#comment-16880361
]
Pierre Zemb commented on HBASE-22618:
-
Cool, I will make the implementation based on master then. Thanks for the tips!
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load balancer use
> [pluggable cost functions instead of this static list of cost
> functions|[https://github.com/apache
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880297#comment-16880297
]
Pierre Zemb commented on HBASE-22618:
-
Thank you [~wchevreuil] for your comments!
Master branch corresponds to Hbase 2.X release right? Is it possible to
backport the patch to 1.4? I must admit that my target cluster is in 1.4 :)
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> available RS.
> h2. Other implementations and ideas
> Clay Baenziger proposed this idea on the dev ML:
> {quote}{color:#22}Could it work to have the stochastic load b
[jira] [Comment Edited] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880118#comment-16880118
]
Pierre Zemb edited comment on HBASE-22618 at 7/8/19 8:26 AM:
-
Hi! Thanks [~apurtell] for your comment. I worked a bit on it last week:
* I added the possibility to load a CostFunction
[https://github.com/PierreZ/hbase/commit/9ddf356ee12f0b39ee0d33211834a718f0dd6194]
It can only load a single function for now
* I reimplemented my balancer as a cost function. Moving from a full balancer
to a single cost function was a huge benefit for us, as we just need to
implement
[https://github.com/PierreZ/hbase/commit/ebe2a1501dda4deb150308a3b380de3bef5961ee#diff-53043f78e2be40cfbf3ff4344bb30bd0R69]
I will now backport my tests, and also add the possibility to load multiple
cost functions
was (Author: pierrez):
Hi! Thanks [~apurtell] for your comment. I worked a bit on it last week:
* I added the possibility to load a CostFunction
[here|[https://github.com/PierreZ/hbase/commit/9ddf356ee12f0b39ee0d33211834a718f0dd6194]]
It can only load a single function for now
* I reimplemented my balancer as a cost function. Moving from a full balancer
to a single cost function was a huge benefit for us, as we just need to
implement
[cost|[https://github.com/PierreZ/hbase/commit/ebe2a1501dda4deb150308a3b380de3bef5961ee#diff-53043f78e2be40cfbf3ff4344bb30bd0R69]]
I will now backport my tests, and also add the possibility to load multiple
cost functions
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, an
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880118#comment-16880118
]
Pierre Zemb commented on HBASE-22618:
-
Hi! Thanks [~apurtell] for your comment. I worked a bit on it last week:
* I added the possibility to load a CostFunction
[here|[https://github.com/PierreZ/hbase/commit/9ddf356ee12f0b39ee0d33211834a718f0dd6194].]
It can only load a single function for now
* I reimplemented my balancer as a cost function. Moving from a full balancer
to a single cost function was a huge benefit for us, as we just need to
implement [cost
title|[https://github.com/PierreZ/hbase/commit/ebe2a1501dda4deb150308a3b380de3bef5961ee#diff-53043f78e2be40cfbf3ff4344bb30bd0R69]]
I will now backport my tests, and also add the possibility to load multiple
cost functions
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> gro
[jira] [Comment Edited] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880118#comment-16880118
]
Pierre Zemb edited comment on HBASE-22618 at 7/8/19 8:25 AM:
-
Hi! Thanks [~apurtell] for your comment. I worked a bit on it last week:
* I added the possibility to load a CostFunction
[here|[https://github.com/PierreZ/hbase/commit/9ddf356ee12f0b39ee0d33211834a718f0dd6194]]
It can only load a single function for now
* I reimplemented my balancer as a cost function. Moving from a full balancer
to a single cost function was a huge benefit for us, as we just need to
implement
[cost|[https://github.com/PierreZ/hbase/commit/ebe2a1501dda4deb150308a3b380de3bef5961ee#diff-53043f78e2be40cfbf3ff4344bb30bd0R69]]
I will now backport my tests, and also add the possibility to load multiple
cost functions
was (Author: pierrez):
Hi! Thanks [~apurtell] for your comment. I worked a bit on it last week:
* I added the possibility to load a CostFunction
[here|[https://github.com/PierreZ/hbase/commit/9ddf356ee12f0b39ee0d33211834a718f0dd6194].]
It can only load a single function for now
* I reimplemented my balancer as a cost function. Moving from a full balancer
to a single cost function was a huge benefit for us, as we just need to
implement [cost
title|[https://github.com/PierreZ/hbase/commit/ebe2a1501dda4deb150308a3b380de3bef5961ee#diff-53043f78e2be40cfbf3ff4344bb30bd0R69]]
I will now backport my tests, and also add the possibility to load multiple
cost functions
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded wit
[jira] [Commented] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16870472#comment-16870472
]
Pierre Zemb commented on HBASE-22618:
-
Our main consideration for the patch is being able to balance regions according
to the max number of regions per RS. As you said, hardware is evolving, and
this is why we ended-up with different kind of hardware.
We will decommissioned some old hardware at some point, but having the
possibility to have both new hardware and old will ease migration to it!
> Provide a way to have Heterogeneous deployment
> --
>
> Key: HBASE-22618
> URL: https://issues.apache.org/jira/browse/HBASE-22618
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.6, 1.4.11
>Reporter: Pierre Zemb
>Priority: Major
>
> Hi,
> We wouls like to open the discussion about bringing the possibility to have
> regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
> cluster running different kind of hardware.
> h2. Why?
> * Cloud deployments means that we may not be able to have the same hardware
> throughout the years
> * Some tables may need special requirements such as SSD whereas others
> should be using hard-drives
> * {color:#22} {color}*in our usecase*{color:#22}(single table,
> dedicated HBase and Hadoop tuned for our usecase, good key
> distribution){color}*, the number of regions per RS was the real limit for
> us*{color:#22}.{color}
> h2. Our usecase
> We found out that *in our usecase*(single table, dedicated HBase and Hadoop
> tuned for our usecase, good key distribution)*, the number of regions per RS
> was the real limit for us*.
> Over the years, due to historical reasons and also the need to benchmark new
> machines, we ended-up with differents groups of hardware: some servers can
> handle only 180 regions, whereas the biggest can handle more than 900.
> Because of such a difference, we had to disable the LoadBalancing to avoid
> the {{roundRobinAssigmnent}}. We developed some internal tooling which are
> responsible for load balancing regions across RegionServers. That was 1.5
> year ago.
> h2. Our Proof-of-concept
> We did work on a Proof-of-concept
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
> and some early tests
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
>
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
> and
> [here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
> We wrote the balancer for our use-case, which means that:
> * there is one table
> * there is no region-replica
> * good key dispersion
> * there is no regions on master
> A rule file is loaded before balancing. It contains lines of rules. A rule is
> composed of a regexp for hostname, and a limit. For example, we could have:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> RegionServers with hostname matching the first rules will have a limit of
> 200, and the others 50. If there's no match, a default is set.
> Thanks to the rule, we have two informations: the max number of regions for
> this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
> try to balance regions according to their capacity.
> Let's take an example. Let's say that we have 20 RS:
> * 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
> each can handle 200 regions.
> * 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
> each can support 50 regions.
> Based on the following rules:
>
> {quote}rs[0-9] 200
> rs1[0-9] 50
> {quote}
>
> The second group is overloaded, whereas the first group has plenty of space.
> We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
> have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
> understand that the cluster is *full at 48.0%* (1200/2500). Based on this
> information, we will then *try to put all the RegionServers to ~48% of load
> according to the rules.* In this case, it will move regions from the second
> group to the first.
> The balancer will:
> * compute how many regions needs to be moved. In our example, by moving 36
> regions on rs10, we could go from 120.0% to 46.0%
> * select regions with lowest data-locality
> * try to find an appropriate RS for the region. We will take the lowest
> ava
[jira] [Updated] (HBASE-22618) Provide a way to have Heterogeneous deployment
[
https://issues.apache.org/jira/browse/HBASE-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Zemb updated HBASE-22618:
Description:
Hi,
We wouls like to open the discussion about bringing the possibility to have
regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
cluster running different kind of hardware.
h2. Why?
* Cloud deployments means that we may not be able to have the same hardware
throughout the years
* Some tables may need special requirements such as SSD whereas others should
be using hard-drives
* {color:#22} {color}*in our usecase*{color:#22}(single table,
dedicated HBase and Hadoop tuned for our usecase, good key
distribution){color}*, the number of regions per RS was the real limit for
us*{color:#22}.{color}
h2. Our usecase
We found out that *in our usecase*(single table, dedicated HBase and Hadoop
tuned for our usecase, good key distribution)*, the number of regions per RS
was the real limit for us*.
Over the years, due to historical reasons and also the need to benchmark new
machines, we ended-up with differents groups of hardware: some servers can
handle only 180 regions, whereas the biggest can handle more than 900. Because
of such a difference, we had to disable the LoadBalancing to avoid the
{{roundRobinAssigmnent}}. We developed some internal tooling which are
responsible for load balancing regions across RegionServers. That was 1.5 year
ago.
h2. Our Proof-of-concept
We did work on a Proof-of-concept
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
and some early tests
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
and
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
We wrote the balancer for our use-case, which means that:
* there is one table
* there is no region-replica
* good key dispersion
* there is no regions on master
A rule file is loaded before balancing. It contains lines of rules. A rule is
composed of a regexp for hostname, and a limit. For example, we could have:
{quote}rs[0-9] 200
rs1[0-9] 50
{quote}
RegionServers with hostname matching the first rules will have a limit of 200,
and the others 50. If there's no match, a default is set.
Thanks to the rule, we have two informations: the max number of regions for
this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
try to balance regions according to their capacity.
Let's take an example. Let's say that we have 20 RS:
* 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
each can handle 200 regions.
* 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
each can support 50 regions.
Based on the following rules:
{quote}rs[0-9] 200
rs1[0-9] 50
{quote}
The second group is overloaded, whereas the first group has plenty of space.
We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
understand that the cluster is *full at 48.0%* (1200/2500). Based on this
information, we will then *try to put all the RegionServers to ~48% of load
according to the rules.* In this case, it will move regions from the second
group to the first.
The balancer will:
* compute how many regions needs to be moved. In our example, by moving 36
regions on rs10, we could go from 120.0% to 46.0%
* select regions with lowest data-locality
* try to find an appropriate RS for the region. We will take the lowest
available RS.
h2. Other implementations and ideas
Clay Baenziger proposed this idea on the dev ML:
{quote}{color:#22}Could it work to have the stochastic load balancer use
[pluggable cost functions instead of this static list of cost
functions|[https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L198]]?
Then, could this type of a load balancer be implemented simply as a new cost
function which folks could choose to load and mix with the others?{color}
{quote}
{color:#22}I think this could be an interesting way to include
user-functions in the mix. As you know your hardawre and the pattern access,
you can easily know which metrics is important for balancing, for us, it will
only be the number of regions, but we could mix-it with the incoming
writes!{color}
bhup
[jira] [Created] (HBASE-22618) Provide a way to have Heterogeneous deployment
Pierre Zemb created HBASE-22618:
---
Summary: Provide a way to have Heterogeneous deployment
Key: HBASE-22618
URL: https://issues.apache.org/jira/browse/HBASE-22618
Project: HBase
Issue Type: Improvement
Affects Versions: 2.1.6, 1.4.11
Reporter: Pierre Zemb
Hi,
We wouls like to open the discussion about bringing the possibility to have
regions deployed on {color:#22}Heterogeneous deployment{color}, i.e Hbase
cluster running different kind of hardware.
h2. Why?
* Cloud deployments means that we may not be able to have the same hardware
throughout the years
* Some tables may need special requirements such as SSD whereas others should
be using hard-drives
* {color:#22} {color}*in our usecase*{color:#22}(single table,
dedicated HBase and Hadoop tuned for our usecase, good key
distribution){color}*, the number of regions per RS was the real limit for
us*{color:#22}.{color}
h2. Our usecase
We found out that *in our usecase*(single table, dedicated HBase and Hadoop
tuned for our usecase, good key distribution)*, the number of regions per RS
was the real limit for us*.
Over the years, due to historical reasons and also the need to benchmark new
machines, we ended-up with differents groups of hardware: some servers can
handle only 180 regions, whereas the biggest can handle more than 900. Because
of such a difference, we had to disable the LoadBalancing to avoid the
{{roundRobinAssigmnent}}. We developed some internal tooling which are
responsible for load balancing regions across RegionServers. That was 1.5 year
ago.
h2. Our Proof-of-concept
We did work on a Proof-of-concept
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
and some early tests
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/HeterogeneousBalancer.java],
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerBalance.java],
and
[here|https://github.com/PierreZ/hbase/blob/dev/hbase14/balancer/hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestHeterogeneousBalancerRules.java].
We wrote the balancer for our use-case, which means that:
* there is one table
* there is no region-replica
* good key dispersion
* there is no regions on master
A rule file is loaded before balancing. It contains lines of rules. A rule is
composed of a regexp for hostname, and a limit. For example, we could have:
{{rs[0-9] 200 rs1[0-9] 50 }}
RegionServers with hostname matching the first rules will have a limit of 200,
and the others 50. If there's no match, a default is set.
Thanks to the rule, we have two informations: the max number of regions for
this cluster, and the rules for each servers. {{HeterogeneousBalancer}} will
try to balance regions according to their capacity.
Let's take an example. Let's say that we have 20 RS:
* 10 RS, named through {{rs0}} to {{rs9}} loaded with 60 regions each, and
each can handle 200 regions.
* 10 RS, named through {{rs10}} to {{rs19}} loaded with 60 regions each, and
each can support 50 regions.
Based on the following rules:
{{rs[0-9] 200 rs1[0-9] 50 }}
The second group is overloaded, whereas the first group has plenty of space.
We know that we can handle at maximum *2500 regions* (200*10 + 50*10) and we
have currently *1200 regions* (60*20). {{HeterogeneousBalancer}} will
understand that the cluster is *full at 48.0%* (1200/2500). Based on this
information, we will then *try to put all the RegionServers to ~48% of load
according to the rules.* In this case, it will move regions from the second
group to the first.
The balancer will:
* compute how many regions needs to be moved. In our example, by moving 36
regions on rs10, we could go from 120.0% to 46.0%
* select regions with lowest data-locality
* try to find an appropriate RS for the region. We will take the lowest
available RS.
h2. Other implementations and ideas
Clay Baenziger proposed this idea on the dev ML:
{quote}{color:#22}Could it work to have the stochastic load balancer use
[pluggable cost functions instead of this static list of cost
functions|[https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L198]]?
Then, could this type of a load balancer be implemented simply as a new cost
function which folks could choose to load and mix with the others?{color}
{quote}
{color:#22}I think this could be an interesting way to include
user-functions in the mix. As you know your hardawre and the pattern access,
you can easily know which metrics is
