[jira] [Updated] (HBASE-16956) Refactor FavoredNodePlan to use regionNames as keys
[ https://issues.apache.org/jira/browse/HBASE-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thiruvel Thirumoolan updated HBASE-16956: - Attachment: HBASE-16956.master.007.patch > Refactor FavoredNodePlan to use regionNames as keys > --- > > Key: HBASE-16956 > URL: https://issues.apache.org/jira/browse/HBASE-16956 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16956.branch-1.001.patch, > HBASE-16956.master.001.patch, HBASE-16956.master.002.patch, > HBASE-16956.master.003.patch, HBASE-16956.master.004.patch, > HBASE-16956.master.005.patch, HBASE-16956.master.006.patch, > HBASE-16956.master.007.patch > > > We would like to rely on the FNPlan cache whether a region is offline or not. > Sticking to regionNames as keys makes that possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16956) Refactor FavoredNodePlan to use regionNames as keys
[ https://issues.apache.org/jira/browse/HBASE-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669765#comment-15669765 ] Thiruvel Thirumoolan commented on HBASE-16956: -- I have updated the patch - HBASE-16956.master.007.patch, sorry got it confused it with HBASE-16962. > Refactor FavoredNodePlan to use regionNames as keys > --- > > Key: HBASE-16956 > URL: https://issues.apache.org/jira/browse/HBASE-16956 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16956.branch-1.001.patch, > HBASE-16956.master.001.patch, HBASE-16956.master.002.patch, > HBASE-16956.master.003.patch, HBASE-16956.master.004.patch, > HBASE-16956.master.005.patch, HBASE-16956.master.006.patch, > HBASE-16956.master.007.patch > > > We would like to rely on the FNPlan cache whether a region is offline or not. > Sticking to regionNames as keys makes that possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17082) ForeignExceptionUtil isn’t packaged when building shaded protocol with -Pcompile-protobuf
[
https://issues.apache.org/jira/browse/HBASE-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669769#comment-15669769
]
Hudson commented on HBASE-17082:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1961 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1961/])
HBASE-17082 ForeignExceptionUtil isnt packaged when building shaded (stack:
rev 08cb550ccd48d07f5d83c5788f6fd29ebf512e81)
* (edit) hbase-protocol-shaded/pom.xml
Revert "HBASE-17082 ForeignExceptionUtil isnt packaged when building (stack:
rev 0f7a7f475134095eaa348af8fb78047970060ca0)
* (edit) hbase-protocol-shaded/pom.xml
> ForeignExceptionUtil isn’t packaged when building shaded protocol with
> -Pcompile-protobuf
> -
>
> Key: HBASE-17082
> URL: https://issues.apache.org/jira/browse/HBASE-17082
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: 17082_attempted_fix.txt, 17082_attempted_fix2.txt,
> HBASE-17082.nothing.patch, HBASE-17082.nothing.patch,
> HBASE-17082.nothing.patch, HBASE-17082.v0.patch, HBASE-17082.v1.patch,
> patch-unit-hbase-client (after v1.patch).txt, patch-unit-hbase-server (after
> v1.patch).txt
>
>
> The source folder will be replaced from src/main/java to
> project.build.directory/protoc-generated-sources when building shaded
> protocol with -Pcompile-protobuf, but we do not copy the
> ForeignExceptionUtil. So the final jar lacks the ForeignExceptionUtil and it
> causes the test error for hbase-client and hbase-server.
> {noformat}
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:[169,36]
> cannot find symbol
> symbol: class ForeignExceptionUtil
> location: package org.apache.hadoop.hbase.util
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java:[100,36]
> cannot find symbol
> symbol: class ForeignExceptionUtil
> location: package org.apache.hadoop.hbase.util
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:[2144,17]
> cannot find symbol
> symbol: variable ForeignExceptionUtil
> location: class org.apache.hadoop.hbase.regionserver.HRegionServer
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java:[938,32]
> cannot find symbol
> symbol: variable ForeignExceptionUtil
> location: class org.apache.hadoop.hbase.master.MasterRpcServices
> {noformat}
> This bug blocks the patches which are against the hbase-protocol-shaded
> module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15786) Create DBB backed MSLAB pool
[ https://issues.apache.org/jira/browse/HBASE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669789#comment-15669789 ] Anoop Sam John commented on HBASE-15786: bq.We need to work on moving mslab and chunk etc. out to a memory module? Move things like MemStoreLABImpl out of regionserver package? You said in other issue that u will work on that.. If u r busy, I can check it . Let me know.. But not as part of this issue.. This is already big in size. > Create DBB backed MSLAB pool > > > Key: HBASE-15786 > URL: https://issues.apache.org/jira/browse/HBASE-15786 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15786.patch > > > We can make use of MSLAB pool for this off heap memstore. > Right now one can specify the global memstore size (heap size) as a % of max > memory using a config. We will add another config with which one can specify > the global off heap memstore size. This will be exact size not as %. When off > heap memstore in use, we will give this entire area for the MSLAB pool and > that will create off heap chunks. So when cells are added to memstore, the > cell data gets copied into the off heap MSLAB chunk spaces. Note that when > the pool size is not really enough and we need additional chunk creation, we > wont use off heap area for that. We dony want to create so many on demand > DBBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669845#comment-15669845 ] ramkrishna.s.vasudevan commented on HBASE-17049: One question though. The WALPE histogram should give a correct picture on the sync part, correct? Because they are calculated in the postSync API? > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669853#comment-15669853 ] ramkrishna.s.vasudevan commented on HBASE-17085: When I changed the wALPE config I could see that AsyncWAL with this patch applied performs atleast 2K ops/sec more than the default size. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
Charlie Qiangeng Xu created HBASE-17110: --- Summary: Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer Key: HBASE-17110 URL: https://issues.apache.org/jira/browse/HBASE-17110 Project: HBase Issue Type: New Feature Components: Balancer Affects Versions: 1.2.4, 2.0.0 Reporter: Charlie Qiangeng Xu Assignee: Charlie Qiangeng Xu This jira is about an enhancement of simpleLoadBalancer. Here we introduce a new strategy: "bytableOverall" which could be controlled by adding: hbase.master.loadbalance.bytableOverall true We have been using the strategy on our largest cluster for several months. it's proven to be very helpful and stable, especially, the result is quite visible to the users. Here is the reason why it's helpful: When operating large scale clusters(our case), some companies still prefer to use SimpleLoadBalancer due to its simplicity, quick balance plan generation, etc. Current SimpleLoadBalancer has two mode: 1. byTable, which only guarantees that the regions of one table could be uniformly distributed. 2. byCluster, which ignores the distribution within tables and balance the regions all together. If the pressures on different tables are different, the first byTable option is preferable one in most case. Yet, this choice sacrifice the cluster level balance and would cause some servers to have significantly higher load, e.g. 240 regions on server A but 410 regions on server B. Consider this case, a cluster has 3 tables and 4 servers: server A has 3 regions: table1:1, table2:1, table3:1 server B has 3 regions: table1:2, table2:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 0 regions. >From the byTable strategy's perspective, the cluster has already been >perfectly balanced on table level. But a perfect status should be like: server A has 2 regions: table2:1, table3:1 server B has 2 regions: table1:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 2 regions: table1:1, table2:2 And this is what the new mode "byTableOverall" can achieve. Two UTs have been added as well and the last one demonstrates the advantage of the new strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669853#comment-15669853 ] ramkrishna.s.vasudevan edited comment on HBASE-17085 at 11/16/16 8:49 AM: -- When I changed the batchsize config I could see that AsyncWAL with this patch applied performs atleast 2K ops/sec more than the default size. was (Author: ram_krish): When I changed the wALPE config I could see that AsyncWAL with this patch applied performs atleast 2K ops/sec more than the default size. > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-17110: Description: This jira is about an enhancement of simpleLoadBalancer. Here we introduce a new strategy: "bytableOverall" which could be controlled by adding: hbase.master.loadbalance.bytableOverall true We have been using the strategy on our largest cluster for several months. it's proven to be very helpful and stable, especially, the result is quite visible to the users. Here is the reason why it's helpful: When operating large scale clusters(our case), some companies still prefer to use SimpleLoadBalancer due to its simplicity, quick balance plan generation, etc. Current SimpleLoadBalancer has two mode: 1. byTable, which only guarantees that the regions of one table could be uniformly distributed. 2. byCluster, which ignores the distribution within tables and balance the regions all together. If the pressures on different tables are different, the first byTable option is preferable one in most case. Yet, this choice sacrifice the cluster level balance and would cause some servers to have significantly higher load, e.g. 242 regions on server A but 417 regions on server B.(real case stats) Consider this case, a cluster has 3 tables and 4 servers: server A has 3 regions: table1:1, table2:1, table3:1 server B has 3 regions: table1:2, table2:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 0 regions. >From the byTable strategy's perspective, the cluster has already been >perfectly balanced on table level. But a perfect status should be like: server A has 2 regions: table2:1, table3:1 server B has 2 regions: table1:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 2 regions: table1:1, table2:2 And this is what the new mode "byTableOverall" can achieve. Two UTs have been added as well and the last one demonstrates the advantage of the new strategy. was: This jira is about an enhancement of simpleLoadBalancer. Here we introduce a new strategy: "bytableOverall" which could be controlled by adding: hbase.master.loadbalance.bytableOverall true We have been using the strategy on our largest cluster for several months. it's proven to be very helpful and stable, especially, the result is quite visible to the users. Here is the reason why it's helpful: When operating large scale clusters(our case), some companies still prefer to use SimpleLoadBalancer due to its simplicity, quick balance plan generation, etc. Current SimpleLoadBalancer has two mode: 1. byTable, which only guarantees that the regions of one table could be uniformly distributed. 2. byCluster, which ignores the distribution within tables and balance the regions all together. If the pressures on different tables are different, the first byTable option is preferable one in most case. Yet, this choice sacrifice the cluster level balance and would cause some servers to have significantly higher load, e.g. 240 regions on server A but 410 regions on server B. Consider this case, a cluster has 3 tables and 4 servers: server A has 3 regions: table1:1, table2:1, table3:1 server B has 3 regions: table1:2, table2:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 0 regions. >From the byTable strategy's perspective, the cluster has already been >perfectly balanced on table level. But a perfect status should be like: server A has 2 regions: table2:1, table3:1 server B has 2 regions: table1:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 2 regions: table1:1, table2:2 And this is what the new mode "byTableOverall" can achieve. Two UTs have been added as well and the last one demonstrates the advantage of the new strategy. > Add an "Overall Strategy" option(balanced both on table level and server > level) to SimpleLoadBalancer > - > > Key: HBASE-17110 > URL: https://issues.apache.org/jira/browse/HBASE-17110 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 2.0.0, 1.2.4 >Reporter: Charlie Qiangeng Xu >Assignee: Charlie Qiangeng Xu > > This jira is about an enhancement of simpleLoadBalancer. Here we introduce a > new strategy: "bytableOverall" which could be controlled by adding: > > hbase.master.loadbalance.bytableOverall > true > > We have been using the strategy on our largest cluster for several months. > it's proven to be very helpful and stable, especially, the result is quite > visible to the users. > Here is the reason why it's helpful: > When operating large scale clusters(our case), some companies still prefer to >
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-17110: Description: This jira is about an enhancement of simpleLoadBalancer. Here we introduce a new strategy: "bytableOverall" which could be controlled by adding: hbase.master.loadbalance.bytableOverall true We have been using the strategy on our largest cluster for several months. it's proven to be very helpful and stable, especially, the result is quite visible to the users. Here is the reason why it's helpful: When operating large scale clusters(our case), some companies still prefer to use SimpleLoadBalancer due to its simplicity, quick balance plan generation, etc. Current SimpleLoadBalancer has two mode: 1. byTable, which only guarantees that the regions of one table could be uniformly distributed. 2. byCluster, which ignores the distribution within tables and balance the regions all together. If the pressures on different tables are different, the first byTable option is preferable one in most case. Yet, this choice sacrifice the cluster level balance and would cause some servers to have significantly higher load, e.g. 242 regions on server A but 417 regions on server B.(real world stats) Consider this case, a cluster has 3 tables and 4 servers: server A has 3 regions: table1:1, table2:1, table3:1 server B has 3 regions: table1:2, table2:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 0 regions. >From the byTable strategy's perspective, the cluster has already been >perfectly balanced on table level. But a perfect status should be like: server A has 2 regions: table2:1, table3:1 server B has 2 regions: table1:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 2 regions: table1:1, table2:2 And this is what the new mode "byTableOverall" can achieve. Two UTs have been added as well and the last one demonstrates the advantage of the new strategy. was: This jira is about an enhancement of simpleLoadBalancer. Here we introduce a new strategy: "bytableOverall" which could be controlled by adding: hbase.master.loadbalance.bytableOverall true We have been using the strategy on our largest cluster for several months. it's proven to be very helpful and stable, especially, the result is quite visible to the users. Here is the reason why it's helpful: When operating large scale clusters(our case), some companies still prefer to use SimpleLoadBalancer due to its simplicity, quick balance plan generation, etc. Current SimpleLoadBalancer has two mode: 1. byTable, which only guarantees that the regions of one table could be uniformly distributed. 2. byCluster, which ignores the distribution within tables and balance the regions all together. If the pressures on different tables are different, the first byTable option is preferable one in most case. Yet, this choice sacrifice the cluster level balance and would cause some servers to have significantly higher load, e.g. 242 regions on server A but 417 regions on server B.(real case stats) Consider this case, a cluster has 3 tables and 4 servers: server A has 3 regions: table1:1, table2:1, table3:1 server B has 3 regions: table1:2, table2:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 0 regions. >From the byTable strategy's perspective, the cluster has already been >perfectly balanced on table level. But a perfect status should be like: server A has 2 regions: table2:1, table3:1 server B has 2 regions: table1:2, table3:2 server C has 3 regions: table1:3, table2:3, table3:3 server D has 2 regions: table1:1, table2:2 And this is what the new mode "byTableOverall" can achieve. Two UTs have been added as well and the last one demonstrates the advantage of the new strategy. > Add an "Overall Strategy" option(balanced both on table level and server > level) to SimpleLoadBalancer > - > > Key: HBASE-17110 > URL: https://issues.apache.org/jira/browse/HBASE-17110 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 2.0.0, 1.2.4 >Reporter: Charlie Qiangeng Xu >Assignee: Charlie Qiangeng Xu > > This jira is about an enhancement of simpleLoadBalancer. Here we introduce a > new strategy: "bytableOverall" which could be controlled by adding: > > hbase.master.loadbalance.bytableOverall > true > > We have been using the strategy on our largest cluster for several months. > it's proven to be very helpful and stable, especially, the result is quite > visible to the users. > Here is the reason why it's helpful: > When operating large scale clusters(our case), some companies
[jira] [Assigned] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-17012:
--
Assignee: ramkrishna.s.vasudevan
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charlie Qiangeng Xu updated HBASE-17110: Attachment: SimpleBalancerBytableOverall.V1 > Add an "Overall Strategy" option(balanced both on table level and server > level) to SimpleLoadBalancer > - > > Key: HBASE-17110 > URL: https://issues.apache.org/jira/browse/HBASE-17110 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 2.0.0, 1.2.4 >Reporter: Charlie Qiangeng Xu >Assignee: Charlie Qiangeng Xu > Attachments: SimpleBalancerBytableOverall.V1 > > > This jira is about an enhancement of simpleLoadBalancer. Here we introduce a > new strategy: "bytableOverall" which could be controlled by adding: > > hbase.master.loadbalance.bytableOverall > true > > We have been using the strategy on our largest cluster for several months. > it's proven to be very helpful and stable, especially, the result is quite > visible to the users. > Here is the reason why it's helpful: > When operating large scale clusters(our case), some companies still prefer to > use SimpleLoadBalancer due to its simplicity, quick balance plan generation, > etc. Current SimpleLoadBalancer has two mode: > 1. byTable, which only guarantees that the regions of one table could be > uniformly distributed. > 2. byCluster, which ignores the distribution within tables and balance the > regions all together. > If the pressures on different tables are different, the first byTable option > is preferable one in most case. Yet, this choice sacrifice the cluster level > balance and would cause some servers to have significantly higher load, e.g. > 242 regions on server A but 417 regions on server B.(real world stats) > Consider this case, a cluster has 3 tables and 4 servers: > server A has 3 regions: table1:1, table2:1, table3:1 > server B has 3 regions: table1:2, table2:2, table3:2 > server C has 3 regions: table1:3, table2:3, table3:3 > server D has 0 regions. > From the byTable strategy's perspective, the cluster has already been > perfectly balanced on table level. But a perfect status should be like: > server A has 2 regions: table2:1, table3:1 > server B has 2 regions: table1:2, table3:2 > server C has 3 regions: table1:3, table2:3, table3:3 > server D has 2 regions: table1:1, table2:2 > And this is what the new mode "byTableOverall" can achieve. > Two UTs have been added as well and the last one demonstrates the advantage > of the new strategy. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16972) Log more details for Scan#next request when responseTooSlow
[
https://issues.apache.org/jira/browse/HBASE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669871#comment-15669871
]
Yu Li commented on HBASE-16972:
---
Thanks for the confirmation [~ndimiduk], just let me know if any action to
take. :-)
> Log more details for Scan#next request when responseTooSlow
> ---
>
> Key: HBASE-16972
> URL: https://issues.apache.org/jira/browse/HBASE-16972
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 1.2.3, 1.1.7
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16972.patch, HBASE-16972.v2.patch,
> HBASE-16972.v3.patch
>
>
> Currently for if responseTooSlow happens on the scan.next call, we will get
> warn log like below:
> {noformat}
> 2016-10-31 11:43:23,430 WARN
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=60193]
> ipc.RpcServer(2574):
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1477885403428,"responsesize":52,"method":"Scan","param":"scanner_id:
> 11 number_of_rows: 2147483647
> close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true
> track_scan_metrics: false renew:
> false","processingtimems":2,"client":"127.0.0.1:60254","queuetimems":0,"class":"HMaster"}
> {noformat}
> From which we only have a {{scanner_id}} and impossible to know what exactly
> this scan is about, like against which region of which table.
> After this JIRA, we will improve the message to something like below (notice
> the last line):
> {noformat}
> 2016-10-31 11:43:23,430 WARN
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=60193]
> ipc.RpcServer(2574):
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1477885403428,"responsesize":52,"method":"Scan","param":"scanner_id:
> 11 number_of_rows: 2147483647
> close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true
> track_scan_metrics: false renew:
> false","processingtimems":2,"client":"127.0.0.1:60254","queuetimems":0,"class":"HMaster",
> "scandetails":"table: hbase:meta region: hbase:meta,,1.1588230740"}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16973) Revisiting default value for hbase.client.scanner.caching
[
https://issues.apache.org/jira/browse/HBASE-16973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669881#comment-15669881
]
Yu Li commented on HBASE-16973:
---
For 1.1, based on our observation from online, should be the former case...
> Revisiting default value for hbase.client.scanner.caching
> -
>
> Key: HBASE-16973
> URL: https://issues.apache.org/jira/browse/HBASE-16973
> Project: HBase
> Issue Type: Task
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: Scan.next_p999.png
>
>
> We are observing below logs for a long-running scan:
> {noformat}
> 2016-10-30 08:51:41,692 WARN
> [B.defaultRpcServer.handler=50,queue=12,port=16020] ipc.RpcServer:
> (responseTooSlow-LongProcessTime): {"processingtimems":24329,
> "call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)",
> "client":"11.251.157.108:50415","scandetails":"table: ae_product_image
> region: ae_product_image,494:
> ,1476872321454.33171a04a683c4404717c43ea4eb8978.","param":"scanner_id:
> 5333521 number_of_rows: 2147483647
> close_scanner: false next_call_seq: 8 client_handles_partials: true
> client_handles_heartbeats: true",
> "starttimems":1477788677363,"queuetimems":0,"class":"HRegionServer","responsesize":818,"method":"Scan"}
> {noformat}
> From which we found the "number_of_rows" is as big as {{Integer.MAX_VALUE}}
> And we also observed a long filter list on the customized scan. After
> checking application code we confirmed that there's no {{Scan.setCaching}} or
> {{hbase.client.scanner.caching}} setting on client side, so it turns out
> using the default value the caching for Scan will be Integer.MAX_VALUE, which
> is really a big surprise.
> After checking code and commit history, I found it's HBASE-11544 which
> changes {{HConstants.DEFAULT_HBASE_CLIENT_SCANNER_CACHING}} from 100 to
> Integer.MAX_VALUE, and from the release note there I could see below notation:
> {noformat}
> Scan caching default has been changed to Integer.Max_Value
> This value works together with the new maxResultSize value from HBASE-12976
> (defaults to 2MB)
> Results returned from server on basis of size rather than number of rows
> Provides better use of network since row size varies amongst tables
> {noformat}
> And I'm afraid this lacks of consideration of the case of scan with filters,
> which may involve many rows but only return with a small result.
> What's more, we still have below comment/code in {{Scan.java}}
> {code}
> /*
>* -1 means no caching
>*/
> private int caching = -1;
> {code}
> But actually the implementation does not follow (instead of no caching, we
> are caching {{Integer.MAX_VALUE}}...).
> So here I'd like to bring up two points:
> 1. Change back the default value of
> HConstants.DEFAULT_HBASE_CLIENT_SCANNER_CACHING to some small value like 128
> 2. Reenforce the semantic of "no caching"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17049) Find out why AsyncFSWAL issues much more syncs than FSHLog
[ https://issues.apache.org/jira/browse/HBASE-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669890#comment-15669890 ] Duo Zhang commented on HBASE-17049: --- For FSHLog, the sync is done inside DFSOutputStream automatically if the buffer is full so we can not record it with our metrics. > Find out why AsyncFSWAL issues much more syncs than FSHLog > -- > > Key: HBASE-17049 > URL: https://issues.apache.org/jira/browse/HBASE-17049 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang > Fix For: 2.0.0 > > > https://issues.apache.org/jira/browse/HBASE-16890?focusedCommentId=15647590&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15647590 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17110:
--
Description:
This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
new strategy: "bytableOverall" which could be controlled by adding:
{noformat}
hbase.master.loadbalance.bytableOverall
true
{noformat}
We have been using the strategy on our largest cluster for several months. it's
proven to be very helpful and stable, especially, the result is quite visible
to the users.
Here is the reason why it's helpful:
When operating large scale clusters(our case), some companies still prefer to
use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
generation, etc. Current SimpleLoadBalancer has two modes:
1. byTable, which only guarantees that the regions of one table could be
uniformly distributed.
2. byCluster, which ignores the distribution within tables and balance the
regions all together.
If the pressures on different tables are different, the first byTable option is
the preferable one in most case. Yet, this choice sacrifice the cluster level
balance and would cause some servers to have significantly higher load, e.g.
242 regions on server A but 417 regions on server B.(real world stats)
Consider this case, a cluster has 3 tables and 4 servers:
{noformat}
server A has 3 regions: table1:1, table2:1, table3:1
server B has 3 regions: table1:2, table2:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 0 regions.
{noformat}
>From the byTable strategy's perspective, the cluster has already been
>perfectly balanced on table level. But a perfect status should be like:
{noformat}
server A has 2 regions: table2:1, table3:1
server B has 2 regions: table1:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 2 regions: table1:1, table2:2
{noformat}
And this is what the new mode "byTableOverall" can achieve.
Two UTs have been added as well and the last one demonstrates the advantage of
the new strategy.
was:
This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
new strategy: "bytableOverall" which could be controlled by adding:
hbase.master.loadbalance.bytableOverall
true
We have been using the strategy on our largest cluster for several months. it's
proven to be very helpful and stable, especially, the result is quite visible
to the users.
Here is the reason why it's helpful:
When operating large scale clusters(our case), some companies still prefer to
use SimpleLoadBalancer due to its simplicity, quick balance plan generation,
etc. Current SimpleLoadBalancer has two mode:
1. byTable, which only guarantees that the regions of one table could be
uniformly distributed.
2. byCluster, which ignores the distribution within tables and balance the
regions all together.
If the pressures on different tables are different, the first byTable option is
preferable one in most case. Yet, this choice sacrifice the cluster level
balance and would cause some servers to have significantly higher load, e.g.
242 regions on server A but 417 regions on server B.(real world stats)
Consider this case, a cluster has 3 tables and 4 servers:
server A has 3 regions: table1:1, table2:1, table3:1
server B has 3 regions: table1:2, table2:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 0 regions.
>From the byTable strategy's perspective, the cluster has already been
>perfectly balanced on table level. But a perfect status should be like:
server A has 2 regions: table2:1, table3:1
server B has 2 regions: table1:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 2 regions: table1:1, table2:2
And this is what the new mode "byTableOverall" can achieve.
Two UTs have been added as well and the last one demonstrates the advantage of
the new strategy.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is qui
[jira] [Commented] (HBASE-17085) AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync
[ https://issues.apache.org/jira/browse/HBASE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669898#comment-15669898 ] Duo Zhang commented on HBASE-17085: --- For me it does not make any difference... What's the batch size you use? > AsyncFSWAL may issue unnecessary AsyncDFSOutput.sync > > > Key: HBASE-17085 > URL: https://issues.apache.org/jira/browse/HBASE-17085 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17085-v1.patch, HBASE-17085-v2.patch, > HBASE-17085-v2.patch, HBASE-17085.patch > > > The problem is in appendAndSync method, we will issue an AsyncDFSOutput.sync > if syncFutures is not empty. The SyncFutures in syncFutures can only be > removed after an AsyncDFSOutput.sync comes back, so before the > AsyncDFSOutput.sync actually returns, we will always issue an > AsyncDFSOutput.sync after an append even if there is no new sync request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17110:
--
Attachment: HBASE-17110.patch
Correcting patch name, please notice to follow the patch naming convention in
future [~xharlie], thanks.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669917#comment-15669917
]
Yu Li commented on HBASE-17110:
---
[~stack]/[~anoop.hbase]/[~enis]/[~ndimiduk]/[~busbey]/[~mantonov]/[~tedyu] this
is something we're using online and could you take a look at the idea/patch
here and let us know your thoughts? Thanks.
And comments from others are well welcome (Smile).
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17099) Is there a plan to support auth connection by username/password like mysql or redis
[
https://issues.apache.org/jira/browse/HBASE-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen updated HBASE-17099:
Description:
Product managers ask our hbase cluster to support auth connection by
username/password.
{code}
private boolean authorizeConnection() throws IOException {
try {
// If auth method is DIGEST, the token was obtained by the
// real user for the effective user, therefore not required to
// authorize real user. doAs is allowed only for simple or kerberos
// authentication
if (user != null && user.getRealUser() != null
&& (authMethod != AuthMethod.DIGEST)) {
ProxyUsers.authorize(user, this.getHostAddress(), conf);
}
authorize(user, connectionHeader, getHostInetAddress());
metrics.authorizationSuccess();
} catch (AuthorizationException ae) {
if (LOG.isDebugEnabled()) {
LOG.debug("Connection authorization failed: " + ae.getMessage(), ae);
}
metrics.authorizationFailure();
setupResponse(authFailedResponse, authFailedCall,
new AccessDeniedException(ae), ae.getMessage());
responder.doRespond(authFailedCall);
return false;
}
return true;
}
{code}
Whether can add a connectionAuthorrizer in method authorizeConnection of
class RpcServer to auth connection by init the handler from conf.
was:
Product managers ask our hbase cluster to support auth connection by
username/password.
{code}
private boolean authorizeConnection() throws IOException {
try {
// If auth method is DIGEST, the token was obtained by the
// real user for the effective user, therefore not required to
// authorize real user. doAs is allowed only for simple or kerberos
// authentication
if (user != null && user.getRealUser() != null
&& (authMethod != AuthMethod.DIGEST)) {
ProxyUsers.authorize(user, this.getHostAddress(), conf);
}
authorize(user, connectionHeader, getHostInetAddress());
metrics.authorizationSuccess();
} catch (AuthorizationException ae) {
if (LOG.isDebugEnabled()) {
LOG.debug("Connection authorization failed: " + ae.getMessage(), ae);
}
metrics.authorizationFailure();
setupResponse(authFailedResponse, authFailedCall,
new AccessDeniedException(ae), ae.getMessage());
responder.doRespond(authFailedCall);
return false;
}
return true;
}
{code}
Whether can add a connectionAuthorrizer in method authorizeConnection of
class RpcServer to auth connection by init the handler from conf.
Issue Type: Brainstorming (was: Wish)
https://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication
FYI
> Is there a plan to support auth connection by username/password like mysql or
> redis
> ---
>
> Key: HBASE-17099
> URL: https://issues.apache.org/jira/browse/HBASE-17099
> Project: HBase
> Issue Type: Brainstorming
> Components: security
>Reporter: liubangchen
>Priority: Trivial
>
> Product managers ask our hbase cluster to support auth connection by
> username/password.
> {code}
> private boolean authorizeConnection() throws IOException {
> try {
> // If auth method is DIGEST, the token was obtained by the
> // real user for the effective user, therefore not required to
> // authorize real user. doAs is allowed only for simple or kerberos
> // authentication
> if (user != null && user.getRealUser() != null
> && (authMethod != AuthMethod.DIGEST)) {
> ProxyUsers.authorize(user, this.getHostAddress(), conf);
> }
> authorize(user, connectionHeader, getHostInetAddress());
> metrics.authorizationSuccess();
> } catch (AuthorizationException ae) {
> if (LOG.isDebugEnabled()) {
> LOG.debug("Connection authorization failed: " + ae.getMessage(),
> ae);
> }
> metrics.authorizationFailure();
> setupResponse(authFailedResponse, authFailedCall,
> new AccessDeniedException(ae), ae.getMessage());
> responder.doRespond(authFailedCall);
> return false;
> }
> return true;
> }
> {code}
> Whether can add a connectionAuthorrizer in method authorizeConnection of
> class RpcServer to auth connection by init the handler from conf.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17084) Delete HMerge (dead code)
[ https://issues.apache.org/jira/browse/HBASE-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669927#comment-15669927 ] Appy commented on HBASE-17084: -- Thanks for pointing it out [~larsgeorge], let me take the discussion there. > Delete HMerge (dead code) > - > > Key: HBASE-17084 > URL: https://issues.apache.org/jira/browse/HBASE-17084 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Trivial > Attachments: HBASE-17084.master.001.patch > > > HMerge isn't used anywhere. Can we delete it? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669931#comment-15669931
]
Charlie Qiangeng Xu commented on HBASE-17110:
-
Whoops, my bad, thank you for correcting that :)
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charlie Qiangeng Xu updated HBASE-17110:
Description:
This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
new strategy: "bytableOverall" which could be controlled by adding:
{noformat}
hbase.master.loadbalance.bytableOverall
true
{noformat}
We have been using the strategy on our largest cluster for several months. it's
proven to be very helpful and stable, especially, the result is quite visible
to the users.
Here is the reason why it's helpful:
When operating large scale clusters(our case), some companies still prefer to
use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
generation, etc. Current SimpleLoadBalancer has two modes:
1. byTable, which only guarantees that the regions of one table could be
uniformly distributed.
2. byCluster, which ignores the distribution within tables and balance the
regions all together.
If the pressures on different tables are different, the first byTable option is
the preferable one in most case. Yet, this choice sacrifice the cluster level
balance and would cause some servers to have significantly higher load, e.g.
242 regions on server A but 417 regions on server B.(real world stats)
Consider this case, a cluster has 3 tables and 4 servers:
{noformat}
server A has 3 regions: table1:1, table2:1, table3:1
server B has 3 regions: table1:2, table2:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 0 regions.
{noformat}
>From the byTable strategy's perspective, the cluster has already been
>perfectly balanced on table level. But a perfect status should be like:
{noformat}
server A has 2 regions: table2:1, table3:1
server B has 2 regions: table1:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 2 regions: table1:1, table2:2
{noformat}
We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
table2 and table3 still keep balanced.
And this is what the new mode "byTableOverall" can achieve.
Two UTs have been added as well and the last one demonstrates the advantage of
the new strategy.
Also, a onConfigurationChange method has been implemented to hot control the
"slop" variable.
was:
This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
new strategy: "bytableOverall" which could be controlled by adding:
{noformat}
hbase.master.loadbalance.bytableOverall
true
{noformat}
We have been using the strategy on our largest cluster for several months. it's
proven to be very helpful and stable, especially, the result is quite visible
to the users.
Here is the reason why it's helpful:
When operating large scale clusters(our case), some companies still prefer to
use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
generation, etc. Current SimpleLoadBalancer has two modes:
1. byTable, which only guarantees that the regions of one table could be
uniformly distributed.
2. byCluster, which ignores the distribution within tables and balance the
regions all together.
If the pressures on different tables are different, the first byTable option is
the preferable one in most case. Yet, this choice sacrifice the cluster level
balance and would cause some servers to have significantly higher load, e.g.
242 regions on server A but 417 regions on server B.(real world stats)
Consider this case, a cluster has 3 tables and 4 servers:
{noformat}
server A has 3 regions: table1:1, table2:1, table3:1
server B has 3 regions: table1:2, table2:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 0 regions.
{noformat}
>From the byTable strategy's perspective, the cluster has already been
>perfectly balanced on table level. But a perfect status should be like:
{noformat}
server A has 2 regions: table2:1, table3:1
server B has 2 regions: table1:2, table3:2
server C has 3 regions: table1:3, table2:3, table3:3
server D has 2 regions: table1:1, table2:2
{noformat}
And this is what the new mode "byTableOverall" can achieve.
Two UTs have been added as well and the last one demonstrates the advantage of
the new strategy.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadB
[jira] [Commented] (HBASE-16169) Make RegionSizeCalculator scalable
[
https://issues.apache.org/jira/browse/HBASE-16169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670003#comment-15670003
]
Hadoop QA commented on HBASE-16169:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 15s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 159m 37s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839127/HBASE-16169.master.007.patch
|
| JIRA Issue | HBASE-16169 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux d29bb4e92f01 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670010#comment-15670010
]
Guanghao Zhang commented on HBASE-17110:
+1 for this idea. We have a similar implementation on XiaoMi HBase cluster. Can
you upload the patch to review board? Thanks.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16169) Make RegionSizeCalculator scalable
[ https://issues.apache.org/jira/browse/HBASE-16169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670026#comment-15670026 ] Thiruvel Thirumoolan commented on HBASE-16169: -- Thanks [~stack], the recent precommit builds are much better. > Make RegionSizeCalculator scalable > -- > > Key: HBASE-16169 > URL: https://issues.apache.org/jira/browse/HBASE-16169 > Project: HBase > Issue Type: Sub-task > Components: mapreduce, scaling >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16169.master.000.patch, > HBASE-16169.master.001.patch, HBASE-16169.master.002.patch, > HBASE-16169.master.003.patch, HBASE-16169.master.004.patch, > HBASE-16169.master.005.patch, HBASE-16169.master.006.patch, > HBASE-16169.master.007.patch, HBASE-16169.master.007.patch, > HBASE-16169.master.008.patch > > > RegionSizeCalculator is needed for better split generation of MR jobs. This > requires RegionLoad which can be obtained via ClusterStatus, i.e. accessing > Master. We don't want master to be in this path. > The proposal is to add an API to the RegionServer that gets RegionLoad of all > regions hosted on it or those of a table if specified. RegionSizeCalculator > can use the latter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16956) Refactor FavoredNodePlan to use regionNames as keys
[
https://issues.apache.org/jira/browse/HBASE-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670033#comment-15670033
]
Hadoop QA commented on HBASE-16956:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 40s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 1s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839135/HBASE-16956.master.007.patch
|
| JIRA Issue | HBASE-16956 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8127299fefa3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4490/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4490/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Refactor FavoredNodePlan to use regionNames as keys
> ---
>
> Key: HBASE-16956
> URL: https://issues.apache.org/jira/browse/HBASE-16956
> Project: HBase
> Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16956.branch
[jira] [Updated] (HBASE-15806) An endpoint-based export tool
[ https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-15806: -- Status: Open (was: Patch Available) > An endpoint-based export tool > - > > Key: HBASE-15806 > URL: https://issues.apache.org/jira/browse/HBASE-15806 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: Experiment.png, HBASE-15806-v1.patch, > HBASE-15806-v2.patch, HBASE-15806-v3.patch, HBASE-15806.patch, > HBASE-15806.v4.patch, HBASE-15806.v5.patch > > > The time for exporting table can be reduced, if we use the endpoint technique > to export the hdfs files by the region server rather than by hbase client. > In my experiments, the elapsed time of endpoint-based export can be less than > half of current export tool (enable the hdfs compression) > But the shortcomings is we need to alter table for deploying the endpoint > any comments about this? thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16169) Make RegionSizeCalculator scalable
[
https://issues.apache.org/jira/browse/HBASE-16169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670091#comment-15670091
]
Hadoop QA commented on HBASE-16169:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 26s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 165m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839132/HBASE-16169.master.008.patch
|
| JIRA Issue | HBASE-16169 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 26edb332c5d2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-
[jira] [Updated] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-17012:
---
Status: Patch Available (was: Open)
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-17012:
---
Attachment: HBASE-17012_1.patch
Patch for trunk. Works on BBWriter.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17076) implement getAndPut() and getAndDelete()
[
https://issues.apache.org/jira/browse/HBASE-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670102#comment-15670102
]
Hadoop QA commented on HBASE-17076:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 11 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 31s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 39s {color}
| {color:red} hbase-rest in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 179m 1s {color
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670110#comment-15670110
]
ramkrishna.s.vasudevan commented on HBASE-17012:
Oh. Just seeing that EncryptedKVEncoder.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15806) An endpoint-based export tool
[ https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-15806: -- Attachment: HBASE-15806.v6.patch > An endpoint-based export tool > - > > Key: HBASE-15806 > URL: https://issues.apache.org/jira/browse/HBASE-15806 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: Experiment.png, HBASE-15806-v1.patch, > HBASE-15806-v2.patch, HBASE-15806-v3.patch, HBASE-15806.patch, > HBASE-15806.v4.patch, HBASE-15806.v5.patch, HBASE-15806.v6.patch > > > The time for exporting table can be reduced, if we use the endpoint technique > to export the hdfs files by the region server rather than by hbase client. > In my experiments, the elapsed time of endpoint-based export can be less than > half of current export tool (enable the hdfs compression) > But the shortcomings is we need to alter table for deploying the endpoint > any comments about this? thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15806) An endpoint-based export tool
[ https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-15806: -- Status: Patch Available (was: Open) > An endpoint-based export tool > - > > Key: HBASE-15806 > URL: https://issues.apache.org/jira/browse/HBASE-15806 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: Experiment.png, HBASE-15806-v1.patch, > HBASE-15806-v2.patch, HBASE-15806-v3.patch, HBASE-15806.patch, > HBASE-15806.v4.patch, HBASE-15806.v5.patch, HBASE-15806.v6.patch > > > The time for exporting table can be reduced, if we use the endpoint technique > to export the hdfs files by the region server rather than by hbase client. > In my experiments, the elapsed time of endpoint-based export can be less than > half of current export tool (enable the hdfs compression) > But the shortcomings is we need to alter table for deploying the endpoint > any comments about this? thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670116#comment-15670116
]
Anoop Sam John commented on HBASE-17110:
Interesting.. Why you think this can not be an improvement for the byTable
strategy but a new strategy?
I see not a new strategy but a config based extension of byTable strategy.
But my doubt is why such a config needed? Why can't the SimpleLB byTable
strategy do this always? First preference to balance by table and then make
sure the overall balance also achieved. I mean at balance time, even if it sees
by tables it is perfect, don't stop there and try find a more perfect cluster
level balance. Than a new config requirement, we can do it always. I can see
the new config is default OFF. May be on existing patch branches u dont want a
behave change? At least for master, I feel at least the config to be ON by
default.. And IMO the config itself not required. Why we should not allow this
better balancing to work?
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670165#comment-15670165
]
ramkrishna.s.vasudevan commented on HBASE-17012:
I think EncryptedKVEncoder is ok as such. Because even if we
{code}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
OutputStream cout = encryptor.createEncryptionStream(baos);
{code}
wrap the BAOS with ByteBufferWriterOuputStream, still what we write is to the
cout (that is the encrypted stream).
{code}
StreamUtils.writeRawVInt32(cout, KeyValueUtil.keyLength(cell));
StreamUtils.writeRawVInt32(cout, cell.getValueLength());
// To support tags
StreamUtils.writeRawVInt32(cout, tlen);
// Write row, qualifier, and family
StreamUtils.writeRawVInt32(cout, cell.getRowLength());
cout.write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength());
StreamUtils.writeRawVInt32(cout, cell.getFamilyLength());
cout.write(cell.getFamilyArray(), cell.getFamilyOffset(),
cell.getFamilyLength());
{code}
So am not sure how to write the ByteBuffer content directly to cout. But I
could actually use the hbase's BAOS and try writing to it and then do a
getBuffer() and use that buffer to write to cout.
But finally we need that baos to actually write it to the actual outputstream
'out'.
So am not sure if this will be same as writing individual byte[] using cout.
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16972) Log more details for Scan#next request when responseTooSlow
[
https://issues.apache.org/jira/browse/HBASE-16972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670240#comment-15670240
]
Sean Busbey commented on HBASE-16972:
-
{quote}
Strong objections Sean Busbey? I guess it's already out in 1.2.4, so that's
that.
{quote}
Indeed, my objection was not strong.
> Log more details for Scan#next request when responseTooSlow
> ---
>
> Key: HBASE-16972
> URL: https://issues.apache.org/jira/browse/HBASE-16972
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Affects Versions: 1.2.3, 1.1.7
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: HBASE-16972.patch, HBASE-16972.v2.patch,
> HBASE-16972.v3.patch
>
>
> Currently for if responseTooSlow happens on the scan.next call, we will get
> warn log like below:
> {noformat}
> 2016-10-31 11:43:23,430 WARN
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=60193]
> ipc.RpcServer(2574):
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1477885403428,"responsesize":52,"method":"Scan","param":"scanner_id:
> 11 number_of_rows: 2147483647
> close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true
> track_scan_metrics: false renew:
> false","processingtimems":2,"client":"127.0.0.1:60254","queuetimems":0,"class":"HMaster"}
> {noformat}
> From which we only have a {{scanner_id}} and impossible to know what exactly
> this scan is about, like against which region of which table.
> After this JIRA, we will improve the message to something like below (notice
> the last line):
> {noformat}
> 2016-10-31 11:43:23,430 WARN
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=60193]
> ipc.RpcServer(2574):
> (responseTooSlow):
> {"call":"Scan(org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ScanRequest)",
> "starttimems":1477885403428,"responsesize":52,"method":"Scan","param":"scanner_id:
> 11 number_of_rows: 2147483647
> close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true
> track_scan_metrics: false renew:
> false","processingtimems":2,"client":"127.0.0.1:60254","queuetimems":0,"class":"HMaster",
> "scandetails":"table: hbase:meta region: hbase:meta,,1.1588230740"}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17111) Use Apache CLI in SnapshotInfo tool
Appy created HBASE-17111: Summary: Use Apache CLI in SnapshotInfo tool Key: HBASE-17111 URL: https://issues.apache.org/jira/browse/HBASE-17111 Project: HBase Issue Type: Bug Reporter: Appy Assignee: Appy AbstractHBaseTool uses Apache CLI to manage command line options, help, etc. We should use it for all tools. This jira is about changing SnapshotInfo to use the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17111) Use Apache CLI in SnapshotInfo tool
[ https://issues.apache.org/jira/browse/HBASE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17111: - Issue Type: Sub-task (was: Bug) Parent: HBASE-17027 > Use Apache CLI in SnapshotInfo tool > --- > > Key: HBASE-17111 > URL: https://issues.apache.org/jira/browse/HBASE-17111 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17111.master.001.patch > > > AbstractHBaseTool uses Apache CLI to manage command line options, help, etc. > We should use it for all tools. This jira is about changing SnapshotInfo to > use the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17111) Use Apache CLI in SnapshotInfo tool
[ https://issues.apache.org/jira/browse/HBASE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17111: - Attachment: HBASE-17111.master.001.patch > Use Apache CLI in SnapshotInfo tool > --- > > Key: HBASE-17111 > URL: https://issues.apache.org/jira/browse/HBASE-17111 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17111.master.001.patch > > > AbstractHBaseTool uses Apache CLI to manage command line options, help, etc. > We should use it for all tools. This jira is about changing SnapshotInfo to > use the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17111) Use Apache CLI in SnapshotInfo tool
[ https://issues.apache.org/jira/browse/HBASE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-17111: - Status: Patch Available (was: Open) > Use Apache CLI in SnapshotInfo tool > --- > > Key: HBASE-17111 > URL: https://issues.apache.org/jira/browse/HBASE-17111 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-17111.master.001.patch > > > AbstractHBaseTool uses Apache CLI to manage command line options, help, etc. > We should use it for all tools. This jira is about changing SnapshotInfo to > use the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670275#comment-15670275
]
Anoop Sam John commented on HBASE-17012:
bq.cout.write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength());
Here the unconditional cal to getXXXArray() can be avoided? Or else we will
end up in making so many temp byte[] garbage. ( 5 per Cell !!)
> Handle Offheap cells in CompressedKvEncoder
> ---
>
> Key: HBASE-17012
> URL: https://issues.apache.org/jira/browse/HBASE-17012
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17012_1.patch
>
>
> When we deal with off heap cells we will end up copying Cell components on
> heap
> {code}
> public void write(Cell cell) throws IOException {
> .
> write(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength(),
> compression.rowDict);
> write(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength(),
> compression.familyDict);
> write(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength(),
> compression.qualifierDict);
> ..
> out.write(cell.getValueArray(), cell.getValueOffset(),
> cell.getValueLength());
> ...
> {code}
> We need to avoid this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17012) Handle Offheap cells in CompressedKvEncoder
[
https://issues.apache.org/jira/browse/HBASE-17012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670348#comment-15670348
]
Hadoop QA commented on HBASE-17012:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 83m 54s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839150/HBASE-17012_1.patch |
| JIRA Issue | HBASE-17012 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d82879ae2443 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4491/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4491/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Handle Offheap cells in CompressedKvEnc
[jira] [Commented] (HBASE-15806) An endpoint-based export tool
[
https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670398#comment-15670398
]
Hadoop QA commented on HBASE-15806:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 16 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 20s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 1 new protoc errors in hbase-server.
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 12s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 10s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839151/HBASE-15806.v6.patch |
| JIRA Issue | HBASE-15806 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile cc hbaseprotoc |
| uname | Linux 38f3334168c3 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
https://bu
[jira] [Updated] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
[ https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17088: --- Attachment: HBASE-17088-v3.patch Retry. > Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor > > > Key: HBASE-17088 > URL: https://issues.apache.org/jira/browse/HBASE-17088 > Project: HBase > Issue Type: Improvement > Components: rpc >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-17088-v1.patch, HBASE-17088-v2.patch, > HBASE-17088-v3.patch, HBASE-17088-v3.patch > > > 1. The RWQueueRpcExecutor has eight constructor method and the longest one > has ten parameters. But It is only used in SimpleRpcScheduler and easy to > confused when read the code. > 2. There are duplicate method implement in RWQueueRpcExecutor and > BalancedQueueRpcExecutor. They can be implemented in their parent class > RpcExecutor. > 3. SimpleRpcScheduler read many configs to new RpcExecutor. But the > CALL_QUEUE_SCAN_SHARE_CONF_KEY is only needed by RWQueueRpcExecutor. And > CALL_QUEUE_CODEL_TARGET_DELAY, CALL_QUEUE_CODEL_INTERVAL and > CALL_QUEUE_CODEL_LIFO_THRESHOLD are only needed by AdaptiveLifoCoDelCallQueue. > So I thought we can refactor it. Suggestions are welcome. > Review board: https://reviews.apache.org/r/53726/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15786) Create DBB backed MSLAB pool
[ https://issues.apache.org/jira/browse/HBASE-15786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-15786: --- Attachment: HBASE-15786_V2.patch > Create DBB backed MSLAB pool > > > Key: HBASE-15786 > URL: https://issues.apache.org/jira/browse/HBASE-15786 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15786.patch, HBASE-15786_V2.patch > > > We can make use of MSLAB pool for this off heap memstore. > Right now one can specify the global memstore size (heap size) as a % of max > memory using a config. We will add another config with which one can specify > the global off heap memstore size. This will be exact size not as %. When off > heap memstore in use, we will give this entire area for the MSLAB pool and > that will create off heap chunks. So when cells are added to memstore, the > cell data gets copied into the off heap MSLAB chunk spaces. Note that when > the pool size is not really enough and we need additional chunk creation, we > wont use off heap area for that. We dony want to create so many on demand > DBBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17111) Use Apache CLI in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-17111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670531#comment-15670531
]
Hadoop QA commented on HBASE-17111:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 3s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.io.asyncfs.TestSaslFanOutOneBlockAsyncDFSOutput |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839161/HBASE-17111.master.001.patch
|
| JIRA Issue | HBASE-17111 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4edb732a7855 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4493/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4493/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4493/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4493/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatical
[jira] [Commented] (HBASE-15985) clarify promises about edits from replication in ref guide
[ https://issues.apache.org/jira/browse/HBASE-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670539#comment-15670539 ] Phil Yang commented on HBASE-15985: --- Hi, I just see this issue, sorry for late. I am wondering what is the real concern for Increment. If I am not wrong, for Increment, in RS we read the current value and write the sum result to WAL as a Put. So in replication Increment is same as Put, and "at-least-once delivery" will not change the value in peer cluster, right? And for message disordering, we will push timestamp to peer cluster, so the only concern for Increment is in source cluster there are two Increments done in same milliseconds, right? For this case I think we can fix it by preventing using same timestamp from current value to the result. And we have HBASE-9465 now, we can guarantee the order of pushing if we want. Although we may push some continuous logs more than once, it will not break anything because WAL entry is idempotent. > clarify promises about edits from replication in ref guide > -- > > Key: HBASE-15985 > URL: https://issues.apache.org/jira/browse/HBASE-15985 > Project: HBase > Issue Type: Sub-task > Components: documentation, Replication >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15985.1.patch > > > we should make clear in a call out that replication only provides > at-least-once delivery and doesn't guarantee ordering so that e.g. folks > using increments aren't surprised. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
Phil Yang created HBASE-17112: - Summary: Prevent setting timestamp of delta operations being same as previous value's Key: HBASE-17112 URL: https://issues.apache.org/jira/browse/HBASE-17112 Project: HBase Issue Type: Bug Reporter: Phil Yang Assignee: Phil Yang In delta operations, Increment and Append. We will read current value first and then write the new whole result into WAL as the type of Put with current timestamp. If the previous ts is larger than current ts, we will use the previous ts. If we have two Puts with same TS, we will ignore the Put with lower sequence id. It is not friendly with versioning. And for replication we will drop sequence id while writing to peer cluster so in the slave we don't know what the order they are being written. If the pushing is disordered, the result will be wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15985) clarify promises about edits from replication in ref guide
[ https://issues.apache.org/jira/browse/HBASE-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670622#comment-15670622 ] Phil Yang commented on HBASE-15985: --- Open a new issue HBASE-17112 to use a different timestamp. > clarify promises about edits from replication in ref guide > -- > > Key: HBASE-15985 > URL: https://issues.apache.org/jira/browse/HBASE-15985 > Project: HBase > Issue Type: Sub-task > Components: documentation, Replication >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15985.1.patch > > > we should make clear in a call out that replication only provides > at-least-once delivery and doesn't guarantee ordering so that e.g. folks > using increments aren't surprised. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17076) implement getAndPut() and getAndDelete()
[ https://issues.apache.org/jira/browse/HBASE-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670625#comment-15670625 ] ChiaPing Tsai commented on HBASE-17076: --- All tests pass locally > implement getAndPut() and getAndDelete() > > > Key: HBASE-17076 > URL: https://issues.apache.org/jira/browse/HBASE-17076 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17076-v0.patch, HBASE-17076-v1.patch, > HBASE-17076-v2.patch > > > We implement the getAndPut() and getAndDelete() by coprocessor, but there are > a lot of duplicate effort (e.g., data checks, row lock, returned value, and > wal). It is cool if we provide the compare-and-swap primitive. > The draft patch is attached. Any advice and suggestions will be greatly > appreciated. > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17112: -- Attachment: HBASE-17112-v1.patch > Prevent setting timestamp of delta operations being same as previous value's > > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-17112-v1.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17112: -- Affects Version/s: 1.1.7 0.98.23 1.2.4 Status: Patch Available (was: Open) > Prevent setting timestamp of delta operations being same as previous value's > > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4, 0.98.23, 1.1.7 >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-17112-v1.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17112: -- Description: In delta operations, Increment and Append. We will read current value first and then write the new whole result into WAL as the type of Put with current timestamp. If the previous ts is larger than current ts, we will use the previous ts. If we have two Puts with same TS, we will ignore the Put with lower sequence id. It is not friendly with versioning. And for replication we will drop sequence id while writing to peer cluster so in the slave we don't know what the order they are being written. If the pushing is disordered, the result will be wrong. We can set the new ts to previous+1 if the previous is not less than now. was: In delta operations, Increment and Append. We will read current value first and then write the new whole result into WAL as the type of Put with current timestamp. If the previous ts is larger than current ts, we will use the previous ts. If we have two Puts with same TS, we will ignore the Put with lower sequence id. It is not friendly with versioning. And for replication we will drop sequence id while writing to peer cluster so in the slave we don't know what the order they are being written. If the pushing is disordered, the result will be wrong. > Prevent setting timestamp of delta operations being same as previous value's > > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang >Assignee: Phil Yang > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17113:
---
Status: Patch Available (was: Open)
> finding middle key in HFileV2 is always wrong and can cause
> IndexOutOfBoundsException
> --
>
> Key: HBASE-17113
> URL: https://issues.apache.org/jira/browse/HBASE-17113
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.2.4, 0.98.23, 1.1.7, 0.94.17, 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17113-branch-1.patch
>
>
> When we want to split a region, we need to get the middle rowkey from the
> biggest store file.
> Here is the code from HFileBlockIndex.midkey() which help us find a
> approximation middle key.
> {code}
> // Caching, using pread, assuming this is not a compaction.
> HFileBlock midLeafBlock = cachingBlockReader.readBlock(
> midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false,
> true,
> BlockType.LEAF_INDEX, null);
> ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
> int numDataBlocks = b.getInt();
> int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
> int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
> keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
> int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
> + SECONDARY_INDEX_ENTRY_OVERHEAD;
> targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
> {code}
> and in each entry of Non-root block index contains three object:
> 1. Offset of the block referenced by this entry in the file (long)
> 2 .Ondisk size of the referenced block (int)
> 3. RowKey.
> But when we caculating the keyLen from the entry, we forget to take away the
> 12 byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So
> the keyLen is always 12 bytes bigger than the real rowkey length.
> Every time we read the rowkey form the entry, we read 12 bytes from the next
> entry.
> No exception will throw unless the middle key is in the last entry of the
> Non-root block index. which will cause a IndexOutOfBoundsException. That is
> exactly what HBASE-16097 is suffering from.
> {code}
> 2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region
> hitsdb,\x14\x03\x83\x1AX\x1A\x9A
> \x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:756)
> {code}
> It is a quite serious bug. It may exsits from HFileV2 was invented. But no
> one has found out! Since this bug ONLY happens when finding a middlekey, and
> since we compare a rowkey from the left side, adding 12 bytes more to the
> right side is totally OK, no one cares!
> It even won't throw IndexOutOfBoundsException before HBASE-12297. since
> {{Arrays.copyOfRange}} is used, which will check the limit to ensue the
> length won't running past the end of the array.
> But now, {{ByteBufferUtils.toBytes}} is used and IndexOutOfBoundsException
[jira] [Updated] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-17113:
---
Attachment: HBASE-17113-branch-1.patch
> finding middle key in HFileV2 is always wrong and can cause
> IndexOutOfBoundsException
> --
>
> Key: HBASE-17113
> URL: https://issues.apache.org/jira/browse/HBASE-17113
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.17, 2.0.0, 1.1.7, 0.98.23, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17113-branch-1.patch
>
>
> When we want to split a region, we need to get the middle rowkey from the
> biggest store file.
> Here is the code from HFileBlockIndex.midkey() which help us find a
> approximation middle key.
> {code}
> // Caching, using pread, assuming this is not a compaction.
> HFileBlock midLeafBlock = cachingBlockReader.readBlock(
> midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false,
> true,
> BlockType.LEAF_INDEX, null);
> ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
> int numDataBlocks = b.getInt();
> int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
> int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
> keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
> int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
> + SECONDARY_INDEX_ENTRY_OVERHEAD;
> targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
> {code}
> and in each entry of Non-root block index contains three object:
> 1. Offset of the block referenced by this entry in the file (long)
> 2 .Ondisk size of the referenced block (int)
> 3. RowKey.
> But when we caculating the keyLen from the entry, we forget to take away the
> 12 byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So
> the keyLen is always 12 bytes bigger than the real rowkey length.
> Every time we read the rowkey form the entry, we read 12 bytes from the next
> entry.
> No exception will throw unless the middle key is in the last entry of the
> Non-root block index. which will cause a IndexOutOfBoundsException. That is
> exactly what HBASE-16097 is suffering from.
> {code}
> 2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region
> hitsdb,\x14\x03\x83\x1AX\x1A\x9A
> \x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:756)
> {code}
> It is a quite serious bug. It may exsits from HFileV2 was invented. But no
> one has found out! Since this bug ONLY happens when finding a middlekey, and
> since we compare a rowkey from the left side, adding 12 bytes more to the
> right side is totally OK, no one cares!
> It even won't throw IndexOutOfBoundsException before HBASE-12297. since
> {{Arrays.copyOfRange}} is used, which will check the limit to ensue the
> length won't running past the end of the array.
> But now, {{ByteBufferUtils.toBytes}} is used and IndexOutOfBoundsException
[jira] [Commented] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670651#comment-15670651
]
Hadoop QA commented on HBASE-17113:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color}
| {color:red} HBASE-17113 does not apply to branch-1. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839183/HBASE-17113-branch-1.patch
|
| JIRA Issue | HBASE-17113 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4496/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> finding middle key in HFileV2 is always wrong and can cause
> IndexOutOfBoundsException
> --
>
> Key: HBASE-17113
> URL: https://issues.apache.org/jira/browse/HBASE-17113
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.17, 2.0.0, 1.1.7, 0.98.23, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17113-branch-1.patch
>
>
> When we want to split a region, we need to get the middle rowkey from the
> biggest store file.
> Here is the code from HFileBlockIndex.midkey() which help us find a
> approximation middle key.
> {code}
> // Caching, using pread, assuming this is not a compaction.
> HFileBlock midLeafBlock = cachingBlockReader.readBlock(
> midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false,
> true,
> BlockType.LEAF_INDEX, null);
> ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
> int numDataBlocks = b.getInt();
> int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
> int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
> keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
> int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
> + SECONDARY_INDEX_ENTRY_OVERHEAD;
> targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
> {code}
> and in each entry of Non-root block index contains three object:
> 1. Offset of the block referenced by this entry in the file (long)
> 2 .Ondisk size of the referenced block (int)
> 3. RowKey.
> But when we caculating the keyLen from the entry, we forget to take away the
> 12 byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So
> the keyLen is always 12 bytes bigger than the real rowkey length.
> Every time we read the rowkey form the entry, we read 12 bytes from the next
> entry.
> No exception will throw unless the middle key is in the last entry of the
> Non-root block index. which will cause a IndexOutOfBoundsException. That is
> exactly what HBASE-16097 is suffering from.
> {code}
> 2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region
> hitsdb,\x14\x03\x83\x1AX\x1A\x9A
> \x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRe
[jira] [Created] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
Allan Yang created HBASE-17113:
--
Summary: finding middle key in HFileV2 is always wrong and can
cause IndexOutOfBoundsException
Key: HBASE-17113
URL: https://issues.apache.org/jira/browse/HBASE-17113
Project: HBase
Issue Type: Bug
Components: HFile
Affects Versions: 1.2.4, 0.98.23, 1.1.7, 0.94.17, 2.0.0
Reporter: Allan Yang
Assignee: Allan Yang
When we want to split a region, we need to get the middle rowkey from the
biggest store file.
Here is the code from HFileBlockIndex.midkey() which help us find a
approximation middle key.
{code}
// Caching, using pread, assuming this is not a compaction.
HFileBlock midLeafBlock = cachingBlockReader.readBlock(
midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false, true,
BlockType.LEAF_INDEX, null);
ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
int numDataBlocks = b.getInt();
int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
+ SECONDARY_INDEX_ENTRY_OVERHEAD;
targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
{code}
and in each entry of Non-root block index contains three object:
1. Offset of the block referenced by this entry in the file (long)
2 .Ondisk size of the referenced block (int)
3. RowKey.
But when we caculating the keyLen from the entry, we forget to take away the 12
byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So the
keyLen is always 12 bytes bigger than the real rowkey length.
Every time we read the rowkey form the entry, we read 12 bytes from the next
entry.
No exception will throw unless the middle key is in the last entry of the
Non-root block index. which will cause a IndexOutOfBoundsException. That is
exactly what HBASE-16097 is suffering from.
{code}
2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher:
Cache flusher failed for entry [flush region hitsdb,\x14\x03\x83\x1AX\x1A\x9A
\x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkIndex(Buffer.java:532)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
at
org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
at
org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
at
org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
at
org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
at java.lang.Thread.run(Thread.java:756)
{code}
It is a quite serious bug. It may exsits from HFileV2 was invented. But no one
has found out! Since this bug ONLY happens when finding a middlekey, and since
we compare a rowkey from the left side, adding 12 bytes more to the right side
is totally OK, no one cares!
It even won't throw IndexOutOfBoundsException before HBASE-12297. since
{{Arrays.copyOfRange}} is used, which will check the limit to ensue the length
won't running past the end of the array.
But now, {{ByteBufferUtils.toBytes}} is used and IndexOutOfBoundsException
will been thrown.
It happens in our production environment. Because of this bug, the region can't
be split can grow bigger and bigger.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670681#comment-15670681
]
Allan Yang commented on HBASE-17113:
sorry, the issue is duplicated to HBASE-17020, please close it.
> finding middle key in HFileV2 is always wrong and can cause
> IndexOutOfBoundsException
> --
>
> Key: HBASE-17113
> URL: https://issues.apache.org/jira/browse/HBASE-17113
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.17, 2.0.0, 1.1.7, 0.98.23, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17113-branch-1.patch
>
>
> When we want to split a region, we need to get the middle rowkey from the
> biggest store file.
> Here is the code from HFileBlockIndex.midkey() which help us find a
> approximation middle key.
> {code}
> // Caching, using pread, assuming this is not a compaction.
> HFileBlock midLeafBlock = cachingBlockReader.readBlock(
> midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false,
> true,
> BlockType.LEAF_INDEX, null);
> ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
> int numDataBlocks = b.getInt();
> int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
> int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
> keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
> int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
> + SECONDARY_INDEX_ENTRY_OVERHEAD;
> targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
> {code}
> and in each entry of Non-root block index contains three object:
> 1. Offset of the block referenced by this entry in the file (long)
> 2 .Ondisk size of the referenced block (int)
> 3. RowKey.
> But when we caculating the keyLen from the entry, we forget to take away the
> 12 byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So
> the keyLen is always 12 bytes bigger than the real rowkey length.
> Every time we read the rowkey form the entry, we read 12 bytes from the next
> entry.
> No exception will throw unless the middle key is in the last entry of the
> Non-root block index. which will cause a IndexOutOfBoundsException. That is
> exactly what HBASE-16097 is suffering from.
> {code}
> 2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region
> hitsdb,\x14\x03\x83\x1AX\x1A\x9A
> \x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:756)
> {code}
> It is a quite serious bug. It may exsits from HFileV2 was invented. But no
> one has found out! Since this bug ONLY happens when finding a middlekey, and
> since we compare a rowkey from the left side, adding 12 bytes more to the
> right side is totally OK, no one cares!
> It even won't throw IndexOutOfBoundsException before HBASE-12297. since
> {{Arrays.copyOfRange}} is used, which will check the limit to ensue the
> length won't running past the end of the arra
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670683#comment-15670683
]
Yu Li commented on HBASE-17110:
---
Thanks for chiming in [~anoop.hbase].
The reason we make it configurable is that the "more perfect cluster level
balance" is with more precise criterion, and if we turn it on, possibly it'll
move regions more aggressively (but it's true that we could use
{{hbase.regions.slop}} to better control it). Since big batch of region moving
will cause spike and is bad for online performance/stability, we choose to set
the config off by default, meanwhile supplied a shell command to trigger the
cluster level balance manually (and please open another JIRA to upstream the
hbase shell tool fella [~xharlie]). And this "possibly auto big batch of region
moving" is also the reason we're not using {{StochasticLoadBalancer}} online,
FWIW.
For master branch, I agree that we could be more aggressive and make the option
ON by default, so people will know about this option, but better to make a
clear release note about the change.
Let me know your thoughts, thanks.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670696#comment-15670696
]
Yu Li commented on HBASE-17110:
---
And also thanks [~zghaobac] for chiming in. Please open an issue on RB for
easier review [~xharlie], thanks.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17113:
---
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> finding middle key in HFileV2 is always wrong and can cause
> IndexOutOfBoundsException
> --
>
> Key: HBASE-17113
> URL: https://issues.apache.org/jira/browse/HBASE-17113
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.17, 2.0.0, 1.1.7, 0.98.23, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17113-branch-1.patch
>
>
> When we want to split a region, we need to get the middle rowkey from the
> biggest store file.
> Here is the code from HFileBlockIndex.midkey() which help us find a
> approximation middle key.
> {code}
> // Caching, using pread, assuming this is not a compaction.
> HFileBlock midLeafBlock = cachingBlockReader.readBlock(
> midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false,
> true,
> BlockType.LEAF_INDEX, null);
> ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
> int numDataBlocks = b.getInt();
> int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
> int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
> keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
> int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
> + SECONDARY_INDEX_ENTRY_OVERHEAD;
> targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
> {code}
> and in each entry of Non-root block index contains three object:
> 1. Offset of the block referenced by this entry in the file (long)
> 2 .Ondisk size of the referenced block (int)
> 3. RowKey.
> But when we caculating the keyLen from the entry, we forget to take away the
> 12 byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So
> the keyLen is always 12 bytes bigger than the real rowkey length.
> Every time we read the rowkey form the entry, we read 12 bytes from the next
> entry.
> No exception will throw unless the middle key is in the last entry of the
> Non-root block index. which will cause a IndexOutOfBoundsException. That is
> exactly what HBASE-16097 is suffering from.
> {code}
> 2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region
> hitsdb,\x14\x03\x83\x1AX\x1A\x9A
> \x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:756)
> {code}
> It is a quite serious bug. It may exsits from HFileV2 was invented. But no
> one has found out! Since this bug ONLY happens when finding a middlekey, and
> since we compare a rowkey from the left side, adding 12 bytes more to the
> right side is totally OK, no one cares!
> It even won't throw IndexOutOfBoundsException before HBASE-12297. since
> {{Arrays.copyOfRange}} is used, which will check the limit to ensue the
> length won't running past the end of the array.
> But now, {{ByteBufferUtils.toBytes}} is used and I
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670723#comment-15670723 ] Yu Li commented on HBASE-17112: --- bq. And for replication we will drop sequence id while writing to peer cluster so in the slave we don't know what the order they are being written. If the pushing is disordered, the result will be wrong. Makes sense to me, and I think this fault is unacceptable. Also +1 on the straight-forward patch. Let's see what others and HadoopQA will say. > Prevent setting timestamp of delta operations being same as previous value's > > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.7, 0.98.23, 1.2.4 >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-17112-v1.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17113) finding middle key in HFileV2 is always wrong and can cause IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-17113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670733#comment-15670733
]
Yu Li commented on HBASE-17113:
---
yep, small world :-)
> finding middle key in HFileV2 is always wrong and can cause
> IndexOutOfBoundsException
> --
>
> Key: HBASE-17113
> URL: https://issues.apache.org/jira/browse/HBASE-17113
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.94.17, 2.0.0, 1.1.7, 0.98.23, 1.2.4
>Reporter: Allan Yang
>Assignee: Allan Yang
> Attachments: HBASE-17113-branch-1.patch
>
>
> When we want to split a region, we need to get the middle rowkey from the
> biggest store file.
> Here is the code from HFileBlockIndex.midkey() which help us find a
> approximation middle key.
> {code}
> // Caching, using pread, assuming this is not a compaction.
> HFileBlock midLeafBlock = cachingBlockReader.readBlock(
> midLeafBlockOffset, midLeafBlockOnDiskSize, true, true, false,
> true,
> BlockType.LEAF_INDEX, null);
> ByteBuffer b = midLeafBlock.getBufferWithoutHeader();
> int numDataBlocks = b.getInt();
> int keyRelOffset = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 1));
> int keyLen = b.getInt(Bytes.SIZEOF_INT * (midKeyEntry + 2)) -
> keyRelOffset - SECONDARY_INDEX_ENTRY_OVERHEAD;
> int keyOffset = Bytes.SIZEOF_INT * (numDataBlocks + 2) + keyRelOffset
> + SECONDARY_INDEX_ENTRY_OVERHEAD;
> targetMidKey = ByteBufferUtils.toBytes(b, keyOffset, keyLen);
> {code}
> and in each entry of Non-root block index contains three object:
> 1. Offset of the block referenced by this entry in the file (long)
> 2 .Ondisk size of the referenced block (int)
> 3. RowKey.
> But when we caculating the keyLen from the entry, we forget to take away the
> 12 byte overhead(1,2 above, SECONDARY_INDEX_ENTRY_OVERHEAD in the code). So
> the keyLen is always 12 bytes bigger than the real rowkey length.
> Every time we read the rowkey form the entry, we read 12 bytes from the next
> entry.
> No exception will throw unless the middle key is in the last entry of the
> Non-root block index. which will cause a IndexOutOfBoundsException. That is
> exactly what HBASE-16097 is suffering from.
> {code}
> 2016-11-16 05:27:31,991 ERROR [MemStoreFlusher.1]
> regionserver.MemStoreFlusher: Cache flusher failed for entry [flush region
> hitsdb,\x14\x03\x83\x1AX\x1A\x9A
> \x00\x00\x07\x00\x00\x07\x00\x00\x09\x00\x00\x09\x00\x01\x9F\x00F\xE3\x00\x00\x0A\x00\x01~\x00\x00\x08\x00\x5C\x09\x00\x03\x11\x00\xEF\x99,1478311873096.79d3f7f285396b6896f3229e2bcac7af.]
> java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkIndex(Buffer.java:532)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
> at
> org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:529)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1527)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:684)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1976)
> at
> org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7614)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:521)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
> at java.lang.Thread.run(Thread.java:756)
> {code}
> It is a quite serious bug. It may exsits from HFileV2 was invented. But no
> one has found out! Since this bug ONLY happens when finding a middlekey, and
> since we compare a rowkey from the left side, adding 12 bytes more to the
> right side is totally OK, no one cares!
> It even won't throw IndexOutOfBoundsException before HBASE-12297. since
> {{Arrays.copyOfRange}} is used, which will check the limit to ensue the
> length won't running past the end of the array.
> But now, {{ByteBufferUtils.toBytes}} is used and
[jira] [Commented] (HBASE-14960) Fallback to using default RPCControllerFactory if class cannot be loaded
[ https://issues.apache.org/jira/browse/HBASE-14960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670761#comment-15670761 ] Vsevolod Ostapenko commented on HBASE-14960: Hortonworks still ships HBase 1.1.2 in the most recent HDP 2.5 releases. Is it possible to have a patch for this issue on 1.1.2? > Fallback to using default RPCControllerFactory if class cannot be loaded > > > Key: HBASE-14960 > URL: https://issues.apache.org/jira/browse/HBASE-14960 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17 > > Attachments: HBASE-14960-0.98.patch, hbase-14960_v1.patch, > hbase-14960_v2.patch, hbase-14960_v3.patch, hbase-14960_v4.patch > > > In Phoenix + HBase clusters, the hbase-site.xml configuration will point to a > custom rpc controller factory which is a Phoenix-specific one to configure > the priorities for index and system catalog table. > However, sometimes these Phoenix-enabled clusters are used from pure-HBase > client applications resulting in ClassNotFoundExceptions in application code > or MapReduce jobs. Since hbase configuration is shared between > Phoenix-clients and HBase clients, having different configurations at the > client side is hard. > We can instead try to load up the RPCControllerFactory from conf, and if not > found, fallback to the default one (in case this is a pure HBase client). In > case Phoenix is already in the classpath, it will work as usual. > This does not affect the rpc scheduler factory since it is only used at the > server side. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17088) Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
[
https://issues.apache.org/jira/browse/HBASE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670825#comment-15670825
]
Hadoop QA commented on HBASE-17088:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 50s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839169/HBASE-17088-v3.patch |
| JIRA Issue | HBASE-17088 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 49dafd205eee 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4494/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4494/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Refactor RWQueueRpcExecutor/BalancedQueueRpcExecutor/RpcExecutor
>
>
> Key: HBASE-17088
> URL: https://issues.apache.org/jira/browse/HBASE-17088
> Project: HBase
> Issue Type: Improvement
> Components: rpc
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-17088-v1
[jira] [Updated] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15597: Assignee: (was: Sean Busbey) > Clean up configuration keys used in hbase-spark module > -- > > Key: HBASE-15597 > URL: https://issues.apache.org/jira/browse/HBASE-15597 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > This should be considered a blocker for backport to branch-1 since it will > impact our compatibility. > The constants we expose in configuration should all start with "hbase". Since > our configurations keys for the spark integration all relate to that system, > the prefix for all configuration keys (excluding those cases where we need to > do something special due to restrictions in how properties are handled by > e.g. spark) should be "hbase.spark". > Before publishing a public api labeled version of our spark integration we > should review all of our configuration keys to make sure they either conform > to the "hbase.spark" prefix or they have a comment documenting why they need > to be otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14161) Add hbase-spark integration tests to IT jenkins job
[ https://issues.apache.org/jira/browse/HBASE-14161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14161: Assignee: (was: Sean Busbey) > Add hbase-spark integration tests to IT jenkins job > --- > > Key: HBASE-14161 > URL: https://issues.apache.org/jira/browse/HBASE-14161 > Project: HBase > Issue Type: Task > Components: build >Reporter: Sean Busbey > Fix For: 2.0.0 > > > expand the set of ITs we run to include the new hbase-spark tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14375) define public API for spark integration module
[ https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14375: Assignee: (was: Sean Busbey) > define public API for spark integration module > -- > > Key: HBASE-14375 > URL: https://issues.apache.org/jira/browse/HBASE-14375 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > before we can put the spark integration module into a release, we need to > annotate its public api surface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
Yu Li created HBASE-17114:
-
Summary: Add an option to set special retry pause when
encountering CallQueueTooBigException
Key: HBASE-17114
URL: https://issues.apache.org/jira/browse/HBASE-17114
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} instead
of dead-wait. This is good for performance for most cases but might cause a
side-effect that if too many clients connect to the busy, the retry requests
may come over and over again and RS never got the chance for recovering, and
the issue will become especially critical when the target region is META.
So here in this JIRA we propose to supply some special retry pause for CQTBE in
name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
times of {{hbase.client.pause}} default value)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
Arshad Mohammad created HBASE-17115: --- Summary: HMaster/HRegion Info Server does not honour admin.acl Key: HBASE-17115 URL: https://issues.apache.org/jira/browse/HBASE-17115 Project: HBase Issue Type: Bug Reporter: Arshad Mohammad Currently there is no way to enable protected URLs like /jmx, /conf only for admins. This is applicable for both Master and RegionServer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670891#comment-15670891
]
Yu Li commented on HBASE-17114:
---
As it's already late night on my site, will upload the patch tomorrow.
[~eclark] [~mantonov] and [~ghelmling] especially looking forward to your
thoughts on this idea gentlemen. Thanks.
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy, the retry
> requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
> times of {{hbase.client.pause}} default value)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17114:
--
Description:
As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} instead
of dead-wait. This is good for performance for most cases but might cause a
side-effect that if too many clients connect to the busy RS, that the retry
requests may come over and over again and RS never got the chance for
recovering, and the issue will become especially critical when the target
region is META.
So here in this JIRA we propose to supply some special retry pause for CQTBE in
name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
times of {{hbase.client.pause}} default value)
was:
As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} instead
of dead-wait. This is good for performance for most cases but might cause a
side-effect that if too many clients connect to the busy, the retry requests
may come over and over again and RS never got the chance for recovering, and
the issue will become especially critical when the target region is META.
So here in this JIRA we propose to supply some special retry pause for CQTBE in
name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
times of {{hbase.client.pause}} default value)
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy RS, that the
> retry requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
> times of {{hbase.client.pause}} default value)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17115) HMaster/HRegion Info Server does not honour admin.acl
[
https://issues.apache.org/jira/browse/HBASE-17115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670920#comment-15670920
]
Arshad Mohammad commented on HBASE-17115:
-
{{org.apache.hadoop.hbase.http.HttpServer}} has instance of
{{org.apache.hadoop.security.authorize.AccessControlList}} but this is not set
from anywhere.
May be we can introduce one new property hbase.admin.acl in hbase
{code}
/** ACL of who can be admin of HBase web URLs*/
public static final String HBASE_ADMIN_ACL ="hbase.admin.acl";
public static final String DEFAULT_HBASE_ADMIN_ACL = "*";
{code}
and initialize AccessControlList and set in InfoServer
{code}
builder.setACL(new AccessControlList(c.get(
HConstants.HBASE_ADMIN_ACL,
HConstants.DEFAULT_HBASE_ADMIN_ACL)));
{code}
> HMaster/HRegion Info Server does not honour admin.acl
> -
>
> Key: HBASE-17115
> URL: https://issues.apache.org/jira/browse/HBASE-17115
> Project: HBase
> Issue Type: Bug
>Reporter: Arshad Mohammad
>
> Currently there is no way to enable protected URLs like /jmx, /conf only
> for admins. This is applicable for both Master and RegionServer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException
[
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670891#comment-15670891
]
Yu Li edited comment on HBASE-17114 at 11/16/16 4:36 PM:
-
As it's already late night on my site, will upload the patch tomorrow.
[~eclark] [~mantonov] and [~ghelmling] especially looking forward to your
thoughts on this idea gentlemen. Thanks.
And others' thoughts are also welcome (Smile).
was (Author: carp84):
As it's already late night on my site, will upload the patch tomorrow.
[~eclark] [~mantonov] and [~ghelmling] especially looking forward to your
thoughts on this idea gentlemen. Thanks.
> Add an option to set special retry pause when encountering
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}}
> instead of dead-wait. This is good for performance for most cases but might
> cause a side-effect that if too many clients connect to the busy RS, that the
> retry requests may come over and over again and RS never got the chance for
> recovering, and the issue will become especially critical when the target
> region is META.
> So here in this JIRA we propose to supply some special retry pause for CQTBE
> in name of {{hbase.client.pause.special}}, and by default it will be 500ms (5
> times of {{hbase.client.pause}} default value)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15670976#comment-15670976 ] Anoop Sam John commented on HBASE-17112: For append/increment we will use TS as the RS cur time only? I mean even if client passed any TS we will ignore those? A quick read of code gives such an impression? Missing some thing? > Prevent setting timestamp of delta operations being same as previous value's > > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.7, 0.98.23, 1.2.4 >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-17112-v1.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[
https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671047#comment-15671047
]
Hadoop QA commented on HBASE-17112:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 45s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839182/HBASE-17112-v1.patch |
| JIRA Issue | HBASE-17112 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d6ded306b585 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0f7a7f4 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4495/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4495/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4495/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4495/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
[jira] [Commented] (HBASE-17096) checkAndMutateApi doesn't work correctly on 0.98.19+
[
https://issues.apache.org/jira/browse/HBASE-17096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671061#comment-15671061
]
Hudson commented on HBASE-17096:
FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1290 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1290/])
HBASE-17096 checkAndMutateApi doesn't work correctly on 0.98.19+ (apurtell: rev
ccf3108ac15c62164dbf10e03d07e298fd402008)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
> checkAndMutateApi doesn't work correctly on 0.98.19+
>
>
> Key: HBASE-17096
> URL: https://issues.apache.org/jira/browse/HBASE-17096
> Project: HBase
> Issue Type: Bug
>Reporter: Samarth Jain
>Assignee: Heng Chen
> Fix For: 0.98.24
>
> Attachments: HBASE-17096-0.98.patch, HBASE-17096-0.98.v2.patch
>
>
> Below is the test case. It uses some Phoenix APIs for getting hold of admin
> and HConnection but should be easily adopted for an HBase IT test. The second
> checkAndMutate should return false but it is returning true. This test fails
> with HBase-0.98.23 and works fine with HBase-0.98.17
> {code}
> @Test
> public void testCheckAndMutateApi() throws Exception {
> byte[] row = Bytes.toBytes("ROW");
> byte[] tableNameBytes = Bytes.toBytes(generateUniqueName());
> byte[] family = Bytes.toBytes(generateUniqueName());
> byte[] qualifier = Bytes.toBytes("QUALIFIER");
> byte[] oldValue = null;
> byte[] newValue = Bytes.toBytes("VALUE");
> Put put = new Put(row);
> put.add(family, qualifier, newValue);
> try (Connection conn = DriverManager.getConnection(getUrl())) {
> PhoenixConnection phxConn = conn.unwrap(PhoenixConnection.class);
> try (HBaseAdmin admin = phxConn.getQueryServices().getAdmin()) {
> HTableDescriptor tableDesc = new HTableDescriptor(
> TableName.valueOf(tableNameBytes));
> HColumnDescriptor columnDesc = new HColumnDescriptor(family);
> columnDesc.setTimeToLive(120);
> tableDesc.addFamily(columnDesc);
> admin.createTable(tableDesc);
> HTableInterface tableDescriptor =
> admin.getConnection().getTable(tableNameBytes);
> assertTrue(tableDescriptor.checkAndPut(row, family,
> qualifier, oldValue, put));
> Delete delete = new Delete(row);
> RowMutations mutations = new RowMutations(row);
> mutations.add(delete);
> assertTrue(tableDescriptor.checkAndMutate(row, family,
> qualifier, CompareOp.EQUAL, newValue, mutations));
> assertFalse(tableDescriptor.checkAndMutate(row, family,
> qualifier, CompareOp.EQUAL, newValue, mutations));
> }
> }
> }
> {code}
> FYI, [~apurtell], [~jamestaylor], [~lhofhansl].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17093) Enhance SecureBulkLoadClient#secureBulkLoadHFiles() to return the family paths of the final hfiles
[ https://issues.apache.org/jira/browse/HBASE-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-17093: --- Attachment: 17093.v4.txt > Enhance SecureBulkLoadClient#secureBulkLoadHFiles() to return the family > paths of the final hfiles > -- > > Key: HBASE-17093 > URL: https://issues.apache.org/jira/browse/HBASE-17093 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > Attachments: 17093.v3.txt, 17093.v4.txt, 17093.v4.txt > > > Currently SecureBulkLoadClient#secureBulkLoadHFiles() returns boolean value > to indicate success / failure. > Since SecureBulkLoadClient.java is new to master branch, we can change the > return type to be the family paths of the final hfiles. > LoadQueueItem would be moved to hbase-client module. > LoadQueueItem in hbase-server module would delegate to the class in > hbase-client module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16708: - Status: Open (was: Patch Available) > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Attachments: HBASE-16708-v1.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16708) Expose endpoint Coprocessor name in "responseTooSlow" log messages
[ https://issues.apache.org/jira/browse/HBASE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16708: - Status: Patch Available (was: Open) > Expose endpoint Coprocessor name in "responseTooSlow" log messages > -- > > Key: HBASE-16708 > URL: https://issues.apache.org/jira/browse/HBASE-16708 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick Dimiduk >Assignee: Yi Liang > Attachments: HBASE-16708-v1.patch > > > Operational diagnostics of a Phoenix install would be easier if we included > which endpoint coprocessor was being called in this responseTooSlow WARN > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671108#comment-15671108
]
Ted Yu commented on HBASE-17110:
The config should be on by default.
{code}
+private int initialnumRegions;
{code}
Why do we need the above variable ? Can this be deduced from size of hriList ?
{code}
+public BalanceInfo(int nextRegionForUnload, int numRegionsAdded, int
initialnumRegions, List hriList) {
{code}
{code}
+int getinitialnumRegions() {
{code}
Upper case 'i' of initial and 'n' of num - same with variable name.
{code}
+ private List ServerLoadList;
{code}
Lower case the leading 'S'.
{code}
+if (sdv > avgLoadOverall / 10) return true;
{code}
How is the constant of 10 determined ?
{code}
+// But if we need to balanceoverall, we should skip assigning those
regions here and assign them along with other
+// regions( that would be peeled off from RS that has max num of regions)
in the following needsBalanceOverall logic
{code}
Wrap long line.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110.patch, SimpleBalancerBytableOverall.V1
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17108) ZKConfig.getZKQuorumServersString does not return the correct client port number
[ https://issues.apache.org/jira/browse/HBASE-17108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17108: --- Attachment: HBASE-17108-0.98.patch Tests look good Here's a patch that brings 0.98's ZKConfig up to branch-1, fixes the bug in quorum string construction as an intended effect of that, and doesn't present a compatibility problem (ZKConfig is Private). I'll commit later today unless objection. /cc [~lhofhansl] > ZKConfig.getZKQuorumServersString does not return the correct client port > number > > > Key: HBASE-17108 > URL: https://issues.apache.org/jira/browse/HBASE-17108 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Andrew Purtell > Fix For: 0.98.24 > > Attachments: HBASE-17108-0.98.patch > > > ZKConfig.getZKQuorumServersString may not return the correct client port > number, at least on 0.98 branch. See PHOENIX-3485. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14375) define public API for spark integration module
[ https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671117#comment-15671117 ] Sean Busbey commented on HBASE-14375: - Included in this issue: version(s) of spark we support, version(s) of scala we support. Does "support" only cover what we build against by default and provide convenience binaries for or does it include things where the downstream user has to provide compilation options? > define public API for spark integration module > -- > > Key: HBASE-14375 > URL: https://issues.apache.org/jira/browse/HBASE-14375 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > before we can put the spark integration module into a release, we need to > annotate its public api surface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16956) Refactor FavoredNodePlan to use regionNames as keys
[ https://issues.apache.org/jira/browse/HBASE-16956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671151#comment-15671151 ] Thiruvel Thirumoolan commented on HBASE-16956: -- [~devaraj] Precommit passed with latest patch, no flaky tests this time. > Refactor FavoredNodePlan to use regionNames as keys > --- > > Key: HBASE-16956 > URL: https://issues.apache.org/jira/browse/HBASE-16956 > Project: HBase > Issue Type: Sub-task >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16956.branch-1.001.patch, > HBASE-16956.master.001.patch, HBASE-16956.master.002.patch, > HBASE-16956.master.003.patch, HBASE-16956.master.004.patch, > HBASE-16956.master.005.patch, HBASE-16956.master.006.patch, > HBASE-16956.master.007.patch > > > We would like to rely on the FNPlan cache whether a region is offline or not. > Sticking to regionNames as keys makes that possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671154#comment-15671154
]
Andrew Purtell commented on HBASE-17069:
Bisect with good confidence lands here:
bae2a970817d3693946c5b1d4e2382e56b1e94b8 is the first bad commit
{noformat}
commit bae2a970817d3693946c5b1d4e2382e56b1e94b8
Author: stack
Date: Wed Sep 30 11:48:46 2015 -0700
HBASE-14465 Backport 'Allow rowlock to be reader/write' to branch-1
{noformat}
Bisect log:
{noformat}
bad: [a12d0a861db850ded1a66d6be8e3a4a9d2c76a4f] HBASE-16980
TestRowProcessorEndpoint failing consistently (Yu Li)
good: [1a305bb4848ebcda2bd7c0df8f2f9c03ddf5b471] Revert "Set down the client
executor core thread count from 256 in tests: REAPPLY" Revert because missing
JIRA number. Will reapply with JIRA number in a sec.
bad: [79dcc6c0d1deeed1081a0f6b58f23bc201d88308] HBASE-14432 Procedure V2 -
enforce ACL on procedure admin tasks (Stephen Yuan Jiang)
bad: [b7ef854fed3dd147b70395a3b296e1a64ad25ac6] HBASE-13876 Improving
performance of HeapMemoryManager
bad: [e78a7e0806370b03413b29ac24ea81d534067bbc] HBASE-14588 Stop accessing test
resources from within src folder (Andrew Wang)
good: [f0512c340ef12653574250a2a778517a0f69f1d2] HBASE-14513 TestBucketCache
runs obnoxious 1k threads in a unit test
bad: [22c87d9644c600788a0df5456333464cba969c49] HBASE-14563 Disable zombie
TestHFileOutputFormat2
bad: [2445d8595091824fe9bfa3a0243213991ebc722f] HBASE-14519 Purge
TestFavoredNodeAssignmentHelper, a test for an abandoned feature that can hang
bad: [54b64871fff96b35302f4709154eca6aa2699cf7] HBASE-14512 Cache UGI groups
good: [783477c5dc24cfa0a0829c2ff5e974d3b4332b81] HBASE-14518 Give
TestScanEarlyTermination the same treatment as 'HBASE-14378 Get
TestAccessController* passing again...' -- up priority handlers
bad: [bae2a970817d3693946c5b1d4e2382e56b1e94b8] HBASE-14465 Backport 'Allow
rowlock to be reader/write' to branch-1
good: [1f5663a7f560a222981db2345e2035f749f62f9b] HBASE-14230 replace reflection
in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
first bad commit: [bae2a970817d3693946c5b1d4e2382e56b1e94b8] HBASE-14465
Backport 'Allow rowlock to be reader/write' to branch-1
{noformat}
I will now double check this with a retest at the point of this commit and then
the immediate preceding one.
If this is the culprit we will have an interesting one here [~stack]
/cc [~lhofhansl]
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFacto
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671207#comment-15671207
]
stack commented on HBASE-17069:
---
Yes [~apurtell] I'm on board figuring the issue in this commit. We don't want
to undo the general approach. Could be issue in original. Could be in the
backport. Thanks for the work sir.
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, f
[jira] [Commented] (HBASE-14375) define public API for spark integration module
[ https://issues.apache.org/jira/browse/HBASE-14375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671215#comment-15671215 ] Andrew Purtell commented on HBASE-14375: bq. version(s) of spark we support, version(s) of scala we support >From a cursory examination, there are two "latest" Spark releases I would >expect to be necessary to support: 1.6.3 and 2.0.2. Searching on Maven I see >they've pushed up two sets of artifacts for 2.0.2: >org.apache.spark:_2.10: and >org.apache.spark:_2.11: Even if we are only supporting the latest Spark, we have two versions of Scala to build and link against. > define public API for spark integration module > -- > > Key: HBASE-14375 > URL: https://issues.apache.org/jira/browse/HBASE-14375 > Project: HBase > Issue Type: Task > Components: spark >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > > before we can put the spark integration module into a release, we need to > annotate its public api surface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17082) ForeignExceptionUtil isn’t packaged when building shaded protocol with -Pcompile-protobuf
[
https://issues.apache.org/jira/browse/HBASE-17082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671243#comment-15671243
]
stack commented on HBASE-17082:
---
Sorry [~chia7712] The README in hbase-protocol-shaded already states you need
to do 'install' step with -Dcompile-protobuf (Others talk of compile because
that is all they need which is confusing but the way it is for now). Am I
misunderstanding? Thanks.
> ForeignExceptionUtil isn’t packaged when building shaded protocol with
> -Pcompile-protobuf
> -
>
> Key: HBASE-17082
> URL: https://issues.apache.org/jira/browse/HBASE-17082
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: 17082_attempted_fix.txt, 17082_attempted_fix2.txt,
> HBASE-17082.nothing.patch, HBASE-17082.nothing.patch,
> HBASE-17082.nothing.patch, HBASE-17082.v0.patch, HBASE-17082.v1.patch,
> patch-unit-hbase-client (after v1.patch).txt, patch-unit-hbase-server (after
> v1.patch).txt
>
>
> The source folder will be replaced from src/main/java to
> project.build.directory/protoc-generated-sources when building shaded
> protocol with -Pcompile-protobuf, but we do not copy the
> ForeignExceptionUtil. So the final jar lacks the ForeignExceptionUtil and it
> causes the test error for hbase-client and hbase-server.
> {noformat}
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:[169,36]
> cannot find symbol
> symbol: class ForeignExceptionUtil
> location: package org.apache.hadoop.hbase.util
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java:[100,36]
> cannot find symbol
> symbol: class ForeignExceptionUtil
> location: package org.apache.hadoop.hbase.util
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:[2144,17]
> cannot find symbol
> symbol: variable ForeignExceptionUtil
> location: class org.apache.hadoop.hbase.regionserver.HRegionServer
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java:[938,32]
> cannot find symbol
> symbol: variable ForeignExceptionUtil
> location: class org.apache.hadoop.hbase.master.MasterRpcServices
> {noformat}
> This bug blocks the patches which are against the hbase-protocol-shaded
> module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations being same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671261#comment-15671261 ] Ted Yu commented on HBASE-17112: Test failures in TestHRegion are reproducible. > Prevent setting timestamp of delta operations being same as previous value's > > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.7, 0.98.23, 1.2.4 >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-17112-v1.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16998) [Master] Analyze table use reports and update quota violations
[ https://issues.apache.org/jira/browse/HBASE-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser reassigned HBASE-16998: -- Assignee: Josh Elser > [Master] Analyze table use reports and update quota violations > -- > > Key: HBASE-16998 > URL: https://issues.apache.org/jira/browse/HBASE-16998 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > > Given the collected table usage reports from RegionServers, the Master needs > to inspect all filesystem-use quotas and determine which need to move into > violation and which need to move out of violation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-1346) Split column names using a delimeter other than space for TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-1346: - Labels: n (was: ) > Split column names using a delimeter other than space for TableInputFormat > --- > > Key: HBASE-1346 > URL: https://issues.apache.org/jira/browse/HBASE-1346 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Justin Becker > Labels: beginner > > Split column names using a delimiter other than space for TableInputFormat. > The configure(JobConf) method currently splits column names by the space > character. This prevents scanning by columns where the qualifier contains a > space. For example, "myColumn:some key". To be consistent with the shell > maybe allow the following syntax "['myColumn:some key','myOtherColumn:key']" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-1346) Split column names using a delimeter other than space for TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-1346: - Labels: newbie (was: n) > Split column names using a delimeter other than space for TableInputFormat > --- > > Key: HBASE-1346 > URL: https://issues.apache.org/jira/browse/HBASE-1346 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Justin Becker > Labels: beginner > > Split column names using a delimiter other than space for TableInputFormat. > The configure(JobConf) method currently splits column names by the space > character. This prevents scanning by columns where the qualifier contains a > space. For example, "myColumn:some key". To be consistent with the shell > maybe allow the following syntax "['myColumn:some key','myOtherColumn:key']" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-1346) Split column names using a delimeter other than space for TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-1346: - Affects Version/s: 2.0.0 > Split column names using a delimeter other than space for TableInputFormat > --- > > Key: HBASE-1346 > URL: https://issues.apache.org/jira/browse/HBASE-1346 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Justin Becker > Labels: beginner > > Split column names using a delimiter other than space for TableInputFormat. > The configure(JobConf) method currently splits column names by the space > character. This prevents scanning by columns where the qualifier contains a > space. For example, "myColumn:some key". To be consistent with the shell > maybe allow the following syntax "['myColumn:some key','myOtherColumn:key']" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-1346) Split column names using a delimeter other than space for TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-1346: - Labels: beginner (was: newbie) > Split column names using a delimeter other than space for TableInputFormat > --- > > Key: HBASE-1346 > URL: https://issues.apache.org/jira/browse/HBASE-1346 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Justin Becker > Labels: beginner > > Split column names using a delimiter other than space for TableInputFormat. > The configure(JobConf) method currently splits column names by the space > character. This prevents scanning by columns where the qualifier contains a > space. For example, "myColumn:some key". To be consistent with the shell > maybe allow the following syntax "['myColumn:some key','myOtherColumn:key']" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-1346) Split column names using a delimeter other than space for TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-1346: - Affects Version/s: (was: 0.19.1) > Split column names using a delimeter other than space for TableInputFormat > --- > > Key: HBASE-1346 > URL: https://issues.apache.org/jira/browse/HBASE-1346 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Justin Becker > Labels: beginner > > Split column names using a delimiter other than space for TableInputFormat. > The configure(JobConf) method currently splits column names by the space > character. This prevents scanning by columns where the qualifier contains a > space. For example, "myColumn:some key". To be consistent with the shell > maybe allow the following syntax "['myColumn:some key','myOtherColumn:key']" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-1346) Split column names using a delimeter other than space for TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-1346: - Component/s: mapreduce > Split column names using a delimeter other than space for TableInputFormat > --- > > Key: HBASE-1346 > URL: https://issues.apache.org/jira/browse/HBASE-1346 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 2.0.0 >Reporter: Justin Becker > Labels: beginner > > Split column names using a delimiter other than space for TableInputFormat. > The configure(JobConf) method currently splits column names by the space > character. This prevents scanning by columns where the qualifier contains a > space. For example, "myColumn:some key". To be consistent with the shell > maybe allow the following syntax "['myColumn:some key','myOtherColumn:key']" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16025) Cache table state to reduce load on META
[ https://issues.apache.org/jira/browse/HBASE-16025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16025: -- Priority: Blocker (was: Critical) > Cache table state to reduce load on META > > > Key: HBASE-16025 > URL: https://issues.apache.org/jira/browse/HBASE-16025 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Blocker > Fix For: 2.0.0 > > > HBASE-12035 moved keeping table enabled/disabled state from ZooKeeper into > hbase:meta. When we retry operations on the client, we check table state in > order to return a specific message if the table is disabled. This means that > in master we will be going back to meta for every retry, even if a region's > location has not changed. This is going to cause performance issues when a > cluster is already loaded, ie. in cases where regionservers may be returning > CallQueueTooBigException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-2213) HCD should only have those fields explicitly set by user while creating tables
[ https://issues.apache.org/jira/browse/HBASE-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez resolved HBASE-2213. -- Resolution: Won't Fix Stale, re-open if you consider this needs to be implemented. > HCD should only have those fields explicitly set by user while creating tables > -- > > Key: HBASE-2213 > URL: https://issues.apache.org/jira/browse/HBASE-2213 > Project: HBase > Issue Type: Bug >Affects Versions: 0.20.3 >Reporter: ryan rawson > > right now we take the default HCD fields and 'snapshot' them into every HCD. > So things like 'BLOCKCACHE' and 'FILESIZE' are in every table, even if they > don't differ from the defaults. If the default changes in a > meanful/important way, the user is left with the unenviable task of (a) > determining this happened and (b) actually going through and > disabling/altering the tables to fix it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
