[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583722#comment-17583722 ] Huaxiang Sun commented on HBASE-27129: -- Remove 2.4.13 from the fixed version as it is only in 2.5+. > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-4 > > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583152#comment-17583152 ] Hudson commented on HBASE-27129: Results for branch master [build #663 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/663/]: (/) *{color:green}+1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/663/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/663/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/663/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14 > > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583130#comment-17583130 ] Hudson commented on HBASE-27129: Results for branch branch-2 [build #624 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/624/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/624/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/624/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/624/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/624/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14 > > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583110#comment-17583110 ] Hudson commented on HBASE-27129: Results for branch branch-2.5 [build #192 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/192/]: (x) *{color:red}-1 overall{color}* details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/192/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/192/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (x) {color:red}-1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/192/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/192/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > Fix For: 2.5.0, 3.0.0-alpha-4, 2.4.14 > > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556426#comment-17556426 ] Duo Zhang commented on HBASE-27129: --- OK, I think I misunderstood the requirement... Here you want to set the storage policy for the directories other than the store directory. Make sense. Let me take a look at the PR first. > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556365#comment-17556365 ] tianhang tang commented on HBASE-27129: --- [~zhangduo] Thanks for your question. I think we don't have table level configuration right now. The configuration of hbase.hstore.block.storage.policy also takes effect when the region is initialized and acts on the CF directory. In fact, I proposed this patch because we encountered some problems with our heterogeneous cluster. Some machines are pure SSD disks, and our HDFS environment has a physical isolation function (for example, it is forbidden to allocate blocks across computer rooms, etc., hereinafter referred to as HDFS group feature). We set hstore.block.storage.policy to ALL_SSD, but when the non-CF directory under the region directory needs to write data, because: 1. The default storage policy is HOT 2. Due to the existence of the HDFS group feature at the HDFS level, the machine with SATA disk cannot be written to 3. The only accessible DataNodes are SSD disks cause the write to fail. Therefore, I think setting a region-level storage policy directly is the easiest solution. > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-27129) Add a config that allows us to configure region-level storage policies
[ https://issues.apache.org/jira/browse/HBASE-27129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1783#comment-1783 ] Duo Zhang commented on HBASE-27129: --- What is the difference between region level and table level? > Add a config that allows us to configure region-level storage policies > -- > > Key: HBASE-27129 > URL: https://issues.apache.org/jira/browse/HBASE-27129 > Project: HBase > Issue Type: New Feature >Reporter: tianhang tang >Assignee: tianhang tang >Priority: Major > > > hbase.hregion.block.storage.policy > HOT|ALL_SSD|... > > With this config, we can set region-level storage policies. > > We have this config about CF storage policy: > {code:java} > > hbase.hstore.block.storage.policy > ALL_SSD > > {code} > But in addition to CF, we also have some other path under region path, such > like .splits, > recovered.edits, .tmp . > So i want to add a region-level config which could cover the whole path, and > if you have any other requirements for the cf directory, you can continue to > use the previous hbase.hstore.block.storage.policy to specify them separately. -- This message was sent by Atlassian Jira (v8.20.7#820007)