[jira] [Commented] (HBASE-17158) Avoid deadlock caused by HRegion#doDelta
[
https://issues.apache.org/jira/browse/HBASE-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689222#comment-15689222
]
Hadoop QA commented on HBASE-17158:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 50s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840202/HBASE-17158.v2.patch |
| JIRA Issue | HBASE-17158 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 64a2b690954b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 511398f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4597/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4597/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Avoid deadlock caused by HRegion#doDelta
>
>
> Key: HBASE-17158
> URL: https://issues.apache.org/jira/browse/HBASE-17158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17158.v0.patch, HBASE-17158.v1.patch,
> HBASE-17158.v2.patch
>
>
>
[jira] [Commented] (HBASE-16984) Implement getScanner
[
https://issues.apache.org/jira/browse/HBASE-16984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689215#comment-15689215
]
Hadoop QA commented on HBASE-16984:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 32s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840201/HBASE-16984-v4.patch |
| JIRA Issue | HBASE-16984 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4db12bde700f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 511398f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4596/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4596/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement getScanner
>
>
>
[jira] [Comment Edited] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689060#comment-15689060
]
Charlie Qiangeng Xu edited comment on HBASE-17110 at 11/23/16 6:10 AM:
---
Hi [~zghaobac], thanks for pointing these out.
{quote}
If we decide this is a default strategy, this method seems doesn't need 2
arguments?
{quote}
We indeed don't need two variables for simpleLoadBalancer, but unfortunately
the method is shared by StochasticLoadBalancer and other balancers as well.
Even if StochasticLoadBalancer doesn't need "byTable" anymore, we still at
least should accommodate to some existing customized balancers that some users
may have in place.
{quote}
This config is not necessary if this is default strategy?
{quote}
This config here is for the strategy itself and would be helpful for a power
user.
I deliberately added this one since it provides better control to the threshold
of the cluster level load difference, which, usually is more tolerable than the
table level.
For most of the user, overallSlop is just same as slop by default.
was (Author: xharlie):
Hi Guanghao, thanks for pointing these out.
{quote}
If we decide this is a default strategy, this method seems doesn't need 2
arguments?
{quote}
We indeed don't need two variable for simpleLoadBalancer, but unfortunately the
method is shared by stochasticLoadBalancer
and other balancers as well. Even if stochasticLoadBalancer doesn't need
"byTable" anymore, we still at least should
accommodate to some existing customized balancers that some users may have in
place.
{quote}
This config is not necessary if this is default strategy?
{quote}
This config here is for the strategy itself and would be helpful for a power
user.
I deliberately add this one since it provides better control to the threshold
of the cluster level load difference,
which, usually is more tolerable than the table level.
For most of the user, overallSlop is just same as slop by default.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689060#comment-15689060
]
Charlie Qiangeng Xu commented on HBASE-17110:
-
Hi Guanghao, thanks for pointing these out.
{quote}
If we decide this is a default strategy, this method seems doesn't need 2
arguments?
{quote}
We indeed don't need two variable for simpleLoadBalancer, but unfortunately the
method is shared by stochasticLoadBalancer
and other balancers as well. Even if stochasticLoadBalancer doesn't need
"byTable" anymore, we still at least should
accommodate to some existing customized balancers that some users may have in
place.
{quote}
This config is not necessary if this is default strategy?
{quote}
This config here is for the strategy itself and would be helpful for a power
user.
I deliberately add this one since it provides better control to the threshold
of the cluster level load difference,
which, usually is more tolerable than the table level.
For most of the user, overallSlop is just same as slop by default.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688981#comment-15688981
]
Guanghao Zhang edited comment on HBASE-17110 at 11/23/16 5:31 AM:
--
{code}
protected Map>
getAssignmentsByTable(
+ boolean simpleLoadBalancerOverall, boolean forceByCluster) {
{code}
If we decide this is a default strategy, this method seems doesn't need 2
arguments?
{code}
this.overallSlop = conf.getFloat("hbase.regions.overallSlop", slop);
{code}
This config is not necessary if this is default strategy?
was (Author: zghaobac):
{code}
protected Map>
getAssignmentsByTable(
+ boolean simpleLoadBalancerOverall, boolean forceByCluster) {
{code}
If we decide this is a default strategy, this method seems doesn't need 2
arguments?
{code}
this.overallSlop = conf.getFloat("hbase.regions.overallSlop", slop);
{code}
This config is necessary if this is default strategy?
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688981#comment-15688981
]
Guanghao Zhang commented on HBASE-17110:
{code}
protected Map>
getAssignmentsByTable(
+ boolean simpleLoadBalancerOverall, boolean forceByCluster) {
{code}
If we decide this is a default strategy, this method seems doesn't need 2
arguments?
{code}
this.overallSlop = conf.getFloat("hbase.regions.overallSlop", slop);
{code}
This config is necessary if this is default strategy?
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688976#comment-15688976
]
Hadoop QA commented on HBASE-17110:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 19s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 41s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840190/HBASE-17110-V5.patch |
| JIRA Issue | HBASE-17110 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b82e81eefd0e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2b79ac |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4594/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4594/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng
[jira] [Updated] (HBASE-17158) Avoid deadlock caused by HRegion#doDelta
[
https://issues.apache.org/jira/browse/HBASE-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17158:
--
Status: Open (was: Patch Available)
> Avoid deadlock caused by HRegion#doDelta
>
>
> Key: HBASE-17158
> URL: https://issues.apache.org/jira/browse/HBASE-17158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17158.v0.patch, HBASE-17158.v1.patch,
> HBASE-17158.v2.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> private Result doDelta(Operation op, Mutation mutation, long nonceGroup, long
> nonce,
> boolean returnResults) throws IOException {
> checkReadOnly();
> checkResources();
> checkRow(mutation.getRow(), op.toString());
> checkFamilies(mutation.getFamilyCellMap().keySet());
> this.writeRequestsCount.increment();
> WriteEntry writeEntry = null;
> startRegionOperation(op);
> List results = returnResults? new ArrayList(mutation.size()):
> null;
> RowLock rowLock = getRowLockInternal(mutation.getRow(), false);
> MemstoreSize memstoreSize = new MemstoreSize();
> }
> {code}
> The getRowLockInternal() should be moved inside the try block so that the
> timeout won't cause the lock leak. Otherwise, we will stuck in
> HRegion#doClose when closing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17158) Avoid deadlock caused by HRegion#doDelta
[
https://issues.apache.org/jira/browse/HBASE-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17158:
--
Status: Patch Available (was: Open)
> Avoid deadlock caused by HRegion#doDelta
>
>
> Key: HBASE-17158
> URL: https://issues.apache.org/jira/browse/HBASE-17158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17158.v0.patch, HBASE-17158.v1.patch,
> HBASE-17158.v2.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> private Result doDelta(Operation op, Mutation mutation, long nonceGroup, long
> nonce,
> boolean returnResults) throws IOException {
> checkReadOnly();
> checkResources();
> checkRow(mutation.getRow(), op.toString());
> checkFamilies(mutation.getFamilyCellMap().keySet());
> this.writeRequestsCount.increment();
> WriteEntry writeEntry = null;
> startRegionOperation(op);
> List results = returnResults? new ArrayList(mutation.size()):
> null;
> RowLock rowLock = getRowLockInternal(mutation.getRow(), false);
> MemstoreSize memstoreSize = new MemstoreSize();
> }
> {code}
> The getRowLockInternal() should be moved inside the try block so that the
> timeout won't cause the lock leak. Otherwise, we will stuck in
> HRegion#doClose when closing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17158) Avoid deadlock caused by HRegion#doDelta
[
https://issues.apache.org/jira/browse/HBASE-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChiaPing Tsai updated HBASE-17158:
--
Attachment: HBASE-17158.v2.patch
address the comment from [~yuzhih...@gmail.com]
> Avoid deadlock caused by HRegion#doDelta
>
>
> Key: HBASE-17158
> URL: https://issues.apache.org/jira/browse/HBASE-17158
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE-17158.v0.patch, HBASE-17158.v1.patch,
> HBASE-17158.v2.patch
>
>
> {code:title=HRegion.java|borderStyle=solid}
> private Result doDelta(Operation op, Mutation mutation, long nonceGroup, long
> nonce,
> boolean returnResults) throws IOException {
> checkReadOnly();
> checkResources();
> checkRow(mutation.getRow(), op.toString());
> checkFamilies(mutation.getFamilyCellMap().keySet());
> this.writeRequestsCount.increment();
> WriteEntry writeEntry = null;
> startRegionOperation(op);
> List results = returnResults? new ArrayList(mutation.size()):
> null;
> RowLock rowLock = getRowLockInternal(mutation.getRow(), false);
> MemstoreSize memstoreSize = new MemstoreSize();
> }
> {code}
> The getRowLockInternal() should be moved inside the try block so that the
> timeout won't cause the lock leak. Otherwise, we will stuck in
> HRegion#doClose when closing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16984) Implement getScanner
[ https://issues.apache.org/jira/browse/HBASE-16984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16984: -- Attachment: HBASE-16984-v4.patch Retry. > Implement getScanner > > > Key: HBASE-16984 > URL: https://issues.apache.org/jira/browse/HBASE-16984 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16984-v1.patch, HBASE-16984-v2.patch, > HBASE-16984-v3.patch, HBASE-16984-v4.patch, HBASE-16984-v4.patch, > HBASE-16984.patch > > > It will just return the old ResultScanner and work like the > AsyncPrefetchClientScanner. I think we still need this as we can not do time > consuming work in the ScanObserver introduced in HBASE-16838. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16984) Implement getScanner
[
https://issues.apache.org/jira/browse/HBASE-16984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688950#comment-15688950
]
Hadoop QA commented on HBASE-16984:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 24s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840196/HBASE-16984-v4.patch |
| JIRA Issue | HBASE-16984 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 58f5baf5031b 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 511398f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4595/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4595/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688941#comment-15688941
]
Anoop Sam John commented on HBASE-17072:
I can see we make use of next block's size in blockRange().. This is used for
loading loadOnStartup blocks.
Normal read flow, we make use of readBlockDataInternal() and there we dont have
prefetched next block header usage now.. So always we will do 2 seeks and
read? Do we pass onDiskSizeWithHeaderL a non -1 value from calling places? Am
not sure. Sorry am I missing any thing?
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations the same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688929#comment-15688929 ] Ted Yu commented on HBASE-17112: Not branch-1.3 at the moment. 1.1 and 1.2 are open. > Prevent setting timestamp of delta operations the same as previous value's > -- > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.7, 0.98.23, 1.2.4 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17112-branch-1-v1.patch, > HBASE-17112-branch-1-v1.patch, HBASE-17112-branch-1.1-v1.patch, > HBASE-17112-branch-1.1-v1.patch, HBASE-17112-v1.patch, HBASE-17112-v2.patch, > HBASE-17112-v2.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688870#comment-15688870
]
stack commented on HBASE-14223:
---
Any luck w/ progress on this one [~enis] What you thinking? Maybe I could carry
it home? Thanks boss.
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.0.4, 1.3.1, 1.1.7, 1.2.5, 0.98.24
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
>
[jira] [Commented] (HBASE-17167) Pass mvcc to client when scan
[ https://issues.apache.org/jira/browse/HBASE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688862#comment-15688862 ] Duo Zhang commented on HBASE-17167: --- For now we will not expose it to the user level API. > Pass mvcc to client when scan > - > > Key: HBASE-17167 > URL: https://issues.apache.org/jira/browse/HBASE-17167 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Reporter: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > > For the current implementation, if we use batch or allowPartial when scan, > then the row level atomic can not be guaranteed if we need to restart a scan > in the middle of a record due to region move or something else. > We can return the mvcc used to open scanner to client and client could use > this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations the same as previous value's
[ https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688864#comment-15688864 ] Phil Yang commented on HBASE-17112: --- Sorry for late reply, [~tedyu] will we commit to 1.3 and lower branches? Thanks > Prevent setting timestamp of delta operations the same as previous value's > -- > > Key: HBASE-17112 > URL: https://issues.apache.org/jira/browse/HBASE-17112 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.7, 0.98.23, 1.2.4 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17112-branch-1-v1.patch, > HBASE-17112-branch-1-v1.patch, HBASE-17112-branch-1.1-v1.patch, > HBASE-17112-branch-1.1-v1.patch, HBASE-17112-v1.patch, HBASE-17112-v2.patch, > HBASE-17112-v2.patch > > > In delta operations, Increment and Append. We will read current value first > and then write the new whole result into WAL as the type of Put with current > timestamp. If the previous ts is larger than current ts, we will use the > previous ts. > If we have two Puts with same TS, we will ignore the Put with lower sequence > id. It is not friendly with versioning. And for replication we will drop > sequence id while writing to peer cluster so in the slave we don't know what > the order they are being written. If the pushing is disordered, the result > will be wrong. > We can set the new ts to previous+1 if the previous is not less than now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17168) Use a cell as the start point of a scan internally
Duo Zhang created HBASE-17168: - Summary: Use a cell as the start point of a scan internally Key: HBASE-17168 URL: https://issues.apache.org/jira/browse/HBASE-17168 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang Mainly for restart a scan when error. For now, the logic for reverse scan is not stable. You can see the ConnectionUtils.createClosestRowBefore. we just append some 0xFF to the end of a key. This works for most time but theoretically we could have a row between the given row and the 'closest row before'. So here I propose we use a cell as the start point of a scan internally as we can use createFirstOnRow or createLastOnRow to bettr control the restart point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17167) Pass mvcc to client when scan
[ https://issues.apache.org/jira/browse/HBASE-17167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688855#comment-15688855 ] stack commented on HBASE-17167: --- Yeah. I dislike exposing this internal sequence but we're talking read-only Makes reasoning about where client should read easier to do accounting on. > Pass mvcc to client when scan > - > > Key: HBASE-17167 > URL: https://issues.apache.org/jira/browse/HBASE-17167 > Project: HBase > Issue Type: Sub-task > Components: Client, scan >Reporter: Duo Zhang > Fix For: 2.0.0, 1.4.0 > > > For the current implementation, if we use batch or allowPartial when scan, > then the row level atomic can not be guaranteed if we need to restart a scan > in the middle of a record due to region move or something else. > We can return the mvcc used to open scanner to client and client could use > this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge
[ https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688853#comment-15688853 ] Stephen Yuan Jiang commented on HBASE-16119: The draft patch could be reviewed in https://reviews.apache.org/r/54012/ > Procedure v2 - Reimplement merge > > > Key: HBASE-16119 > URL: https://issues.apache.org/jira/browse/HBASE-16119 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > > use the proc-v2 state machine for merge. also update the logic to have a > single meta-writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17167) Pass mvcc to client when scan
Duo Zhang created HBASE-17167: - Summary: Pass mvcc to client when scan Key: HBASE-17167 URL: https://issues.apache.org/jira/browse/HBASE-17167 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang For the current implementation, if we use batch or allowPartial when scan, then the row level atomic can not be guaranteed if we need to restart a scan in the middle of a record due to region move or something else. We can return the mvcc used to open scanner to client and client could use this mvcc to restart a scan to get row level atomic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17159) Improve the assignment plan when server aborted or creating tables, etc.
[ https://issues.apache.org/jira/browse/HBASE-17159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688840#comment-15688840 ] Yu Li commented on HBASE-17159: --- Seems clear enough, please prepare a patch, thanks. [~xharlie] > Improve the assignment plan when server aborted or creating tables, etc. > > > Key: HBASE-17159 > URL: https://issues.apache.org/jira/browse/HBASE-17159 > Project: HBase > Issue Type: New Feature > Components: Balancer, Region Assignment >Affects Versions: 2.0.0, 1.2.4 >Reporter: Charlie Qiangeng Xu > > When master processing a dead serve or creating a new table, the assignment > plan would be generated based on the roundRobinAssignment method of balancer. > Yet if these operations happen a lot, the cluster would be out of balance > both on table level and server level. Balancer would be triggered and may > cause huge amount of region moves( This is what we observed). > Ideally, the assignment should be able to consider the table or cluster level > balance as well as locality(for the case of dead server). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17159) Improve the assignment plan when server aborted or creating tables, etc.
[
https://issues.apache.org/jira/browse/HBASE-17159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688833#comment-15688833
]
Charlie Qiangeng Xu commented on HBASE-17159:
-
Among the comment for the roundRobinAssignment function in the class of
BaseLoadBalancer.java:
{noformat}
* Currently implemented as a round-robin assignment. Same invariant as load
* balancing, all servers holding floor(avg) or ceiling(avg).
*
* TODO: Use block locations from HDFS to place regions with their blocks
{noformat}
and inside the body of roundRobinAssignment function:
{noformat}
// TODO: instead of retainAssignment() and roundRobinAssignment(), we
should just run the
// normal LB.balancerCluster() with unassignedRegions. We only need to have
a candidate
// generator for AssignRegionAction. The LB will ensure the regions are
mostly local
// and balanced. This should also run fast with fewer number of iterations.
{noformat}
> Improve the assignment plan when server aborted or creating tables, etc.
>
>
> Key: HBASE-17159
> URL: https://issues.apache.org/jira/browse/HBASE-17159
> Project: HBase
> Issue Type: New Feature
> Components: Balancer, Region Assignment
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>
> When master processing a dead serve or creating a new table, the assignment
> plan would be generated based on the roundRobinAssignment method of balancer.
> Yet if these operations happen a lot, the cluster would be out of balance
> both on table level and server level. Balancer would be triggered and may
> cause huge amount of region moves( This is what we observed).
> Ideally, the assignment should be able to consider the table or cluster level
> balance as well as locality(for the case of dead server).
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17160: -- Attachment: HBASE-17160.master.002.patch Retry > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, HBASE-17160.master.002.patch, hbase.png, > minor_hbase.png, untangled_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-17160: --- Reopen for addendum that does more cleanup > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, hbase.png, minor_hbase.png, untangled_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16984) Implement getScanner
[ https://issues.apache.org/jira/browse/HBASE-16984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688817#comment-15688817 ] Duo Zhang commented on HBASE-16984: --- Any concerns [~carp84]? Thanks. > Implement getScanner > > > Key: HBASE-16984 > URL: https://issues.apache.org/jira/browse/HBASE-16984 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16984-v1.patch, HBASE-16984-v2.patch, > HBASE-16984-v3.patch, HBASE-16984-v4.patch, HBASE-16984.patch > > > It will just return the old ResultScanner and work like the > AsyncPrefetchClientScanner. I think we still need this as we can not do time > consuming work in the ScanObserver introduced in HBASE-16838. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688806#comment-15688806
]
Yu Li edited comment on HBASE-17072 at 11/23/16 3:51 AM:
-
Oh, just noticed the UT failure reported by HadoopQA, seems relative and worth
a check. So removing my +1 for now...
was (Author: carp84):
Oh, regarding UT failure reported by HadoopQA, seems relative and worth a check.
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688806#comment-15688806
]
Yu Li commented on HBASE-17072:
---
Oh, regarding UT failure reported by HadoopQA, seems relative and worth a check.
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688811#comment-15688811
]
stack commented on HBASE-17072:
---
Thanks you for the review [~carp84] ... yes it is a good catch by
[~sato_eiichi]. Let me address your comments (Thanks for the review). Also have
to figure the failing unit test.
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16984) Implement getScanner
[ https://issues.apache.org/jira/browse/HBASE-16984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16984: -- Attachment: HBASE-16984-v4.patch Add comments of the implementation for AsyncResultScanner. > Implement getScanner > > > Key: HBASE-16984 > URL: https://issues.apache.org/jira/browse/HBASE-16984 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16984-v1.patch, HBASE-16984-v2.patch, > HBASE-16984-v3.patch, HBASE-16984-v4.patch, HBASE-16984.patch > > > It will just return the old ResultScanner and work like the > AsyncPrefetchClientScanner. I think we still need this as we can not do time > consuming work in the ScanObserver introduced in HBASE-16838. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688800#comment-15688800
]
Yu Li commented on HBASE-17072:
---
Nice catch and analysis!
I wonder the issue here is not only for G1GC but also for CMS. We also observed
high cpu usage when testing NettyRpcServer against YCSB workloadE and I suspect
the root cause is the same. (please also take a look here fella [~aoxiang])
Patch v2 LGTM and I've just left some minor comments in RB, minor enough to
address on commit. So here is my +1 sir [~stack]
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17166) ITBLL fails on master unable to find hbase-protocol-shaded content
[
https://issues.apache.org/jira/browse/HBASE-17166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17166:
--
I pushed it. Its most likely the fix. Haven't confirmed yet (waiting on
hbase-it to come back together up on jenkins... working on it). Thanks for
review [~jmhsieh]
> ITBLL fails on master unable to find hbase-protocol-shaded content
> --
>
> Key: HBASE-17166
> URL: https://issues.apache.org/jira/browse/HBASE-17166
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: HBASE-17166.master.001.patch
>
>
> From an internal rig found by [~jmhsieh] running generator step:
> {code}
> 16/11/22 16:51:17 INFO mapreduce.Job: Task Id :
> attempt_1479833370377_0002_m_00_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:425)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[
https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688762#comment-15688762
]
Hadoop QA commented on HBASE-17165:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
47s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 1s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 24s
{color} | {color:red} hbase-server-jdk1.8.0_111 with JDK v1.8.0_111 generated 3
new + 3 unchanged - 0 fixed = 6 total (was 3) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 3
new + 3 unchanged - 0 fixed = 6 total (was 3) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 52s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 112m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:d6626eb |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840178/HBASE-17165.branch-1.2.001.patch
|
| JIRA Issue | HBASE-17165 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2eda990d9682 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-17146) Reconsider rpc timeout calculation in backup restore operation
[ https://issues.apache.org/jira/browse/HBASE-17146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688749#comment-15688749 ] Ted Yu commented on HBASE-17146: Ran through backup related tests - all passed. > Reconsider rpc timeout calculation in backup restore operation > --- > > Key: HBASE-17146 > URL: https://issues.apache.org/jira/browse/HBASE-17146 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-17146-v1.patch > > > We calculate rpc timeout by multiplying # of regions by single_file_timeout > (60 sec) > For big tables this may become very large timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17166) ITBLL fails on master unable to find hbase-protocol-shaded content
[
https://issues.apache.org/jira/browse/HBASE-17166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688742#comment-15688742
]
Hadoop QA commented on HBASE-17166:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 86m 16s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 120m 11s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840174/HBASE-17166.master.001.patch
|
| JIRA Issue | HBASE-17166 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 99e3bc975f6a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / b2b79ac |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4592/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4592/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> ITBLL fails on master unable to find hbase-protocol-shaded content
> --
>
> Key: HBASE-17166
> URL: https://issues.apache.org/jira/browse/HBASE-17166
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter:
[jira] [Commented] (HBASE-17166) ITBLL fails on master unable to find hbase-protocol-shaded content
[
https://issues.apache.org/jira/browse/HBASE-17166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688728#comment-15688728
]
Jonathan Hsieh commented on HBASE-17166:
I did a quick confirmation that ClientProtos was contained in
hbase-protocol-shaded-*.jar.
lgtm. +1.
> ITBLL fails on master unable to find hbase-protocol-shaded content
> --
>
> Key: HBASE-17166
> URL: https://issues.apache.org/jira/browse/HBASE-17166
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Attachments: HBASE-17166.master.001.patch
>
>
> From an internal rig found by [~jmhsieh] running generator step:
> {code}
> 16/11/22 16:51:17 INFO mapreduce.Job: Task Id :
> attempt_1479833370377_0002_m_00_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:425)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688729#comment-15688729
]
Charlie Qiangeng Xu commented on HBASE-17110:
-
Thanks for reviewing the patch sir, fixed the warning and resubmiting
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charlie Qiangeng Xu updated HBASE-17110:
Status: Patch Available (was: Open)
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 1.2.4, 2.0.0
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charlie Qiangeng Xu updated HBASE-17110:
Status: Open (was: Patch Available)
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 1.2.4, 2.0.0
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17110) Add an "Overall Strategy" option(balanced both on table level and server level) to SimpleLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charlie Qiangeng Xu updated HBASE-17110:
Attachment: HBASE-17110-V5.patch
> Add an "Overall Strategy" option(balanced both on table level and server
> level) to SimpleLoadBalancer
> -
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch,
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
>
> hbase.master.loadbalance.bytableOverall
> true
>
> {noformat}
> We have been using the strategy on our largest cluster for several months.
> it's proven to be very helpful and stable, especially, the result is quite
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan
> generation, etc. Current SimpleLoadBalancer has two modes:
> 1. byTable, which only guarantees that the regions of one table could be
> uniformly distributed.
> 2. byCluster, which ignores the distribution within tables and balance the
> regions all together.
> If the pressures on different tables are different, the first byTable option
> is the preferable one in most case. Yet, this choice sacrifice the cluster
> level balance and would cause some servers to have significantly higher load,
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case, a cluster has 3 tables and 4 servers:
> {noformat}
> server A has 3 regions: table1:1, table2:1, table3:1
> server B has 3 regions: table1:2, table2:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
> server A has 2 regions: table2:1, table3:1
> server B has 2 regions: table1:2, table3:2
> server C has 3 regions: table1:3, table2:3, table3:3
> server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1,
> table2 and table3 still keep balanced.
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the
> "slop" variable.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688704#comment-15688704
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 53s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
43s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 10m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 12s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
52m 47s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 10s
{color} | {color:red} root generated 55 new + 18 unchanged - 1 fixed = 73 total
(was 19) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s
{color} | {color:red} hbase-spark-1.6 generated 18 new + 0 unchanged - 0 fixed
= 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s
{color} | {color:red} hbase-spark-1.6-scala-2.10 generated 18 new + 0 unchanged
- 0 fixed = 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s
{color} | {color:red} hbase-spark-scala-2.10 generated 18 new + 0 unchanged - 0
fixed = 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 1m 5s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 30s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 125m 29s
{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-assembly in the patch passed. {color}
[jira] [Updated] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-10676:
--
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Resolving as duplicate/subsumed by HBASE-17072 which purges the ThreadLocal.
Thank you for your hard work in here [~zhaojianbo]... Sorry it took us a while
to get around to this.
> Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
> perforamce of scan
>
>
> Key: HBASE-10676
> URL: https://issues.apache.org/jira/browse/HBASE-10676
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.99.0
>Reporter: zhaojianbo
>Assignee: zhaojianbo
> Attachments: HBASE-10676-0.98-branch-AtomicReferenceV2.patch,
> HBASE-10676-0.98-branchV2.patch
>
>
> PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
> backward seek operation as the comment said:
> {quote}
> we will not incur a backward seek operation if we have already read this
> block's header as part of the previous read's look-ahead. And we also want to
> skip reading the header again if it has already been read.
> {quote}
> But that is not the case. In the code of 0.98, prefetchedHeader is
> threadlocal for one storefile reader, and in the RegionScanner
> lifecycle，different rpc handlers will serve scan requests of the same
> scanner. Even though one handler of previous scan call prefetched the next
> block header, the other handlers of current scan call will still trigger a
> backward seek operation. The process is like this:
> # rs handler1 serves the scan call, reads block1 and prefetches the header of
> block2
> # rs handler2 serves the same scanner's next scan call, because rs handler2
> doesn't know the header of block2 already prefetched by rs handler1, triggers
> a backward seek and reads block2, and prefetches the header of block3.
> It is not the sequential read. So I think that the threadlocal is useless,
> and should be abandoned. I did the work, and evaluated the performance of one
> client, two client and four client scanning the same region with one
> storefile. The test environment is
> # A hdfs cluster with a namenode, a secondary namenode , a datanode in a
> machine
> # A hbase cluster with a zk, a master, a regionserver in the same machine
> # clients are also in the same machine.
> So all the data is local. The storefile is about 22.7GB from our online data,
> 18995949 kvs. Caching is set 1000. And setCacheBlocks(false)
> With the improvement, the client total scan time decreases 21% for the one
> client case, 11% for the two clients case. But the four clients case is
> almost the same. The details tests' data is the following:
> ||case||client||time(ms)||
> | original | 1 | 306222 |
> | new | 1 | 241313 |
> | original | 2 | 416390 |
> | new | 2 | 369064 |
> | original | 4 | 555986 |
> | new | 4 | 562152 |
> With some modification(see the comments below), the newest result is
> ||case||client||time(ms)||case||client||time(ms)||case||client||time(ms)||
> |original|1|306222|new with synchronized|1|239510|new with
> AtomicReference|1|241243|
> |original|2|416390|new with synchronized|2|365367|new with
> AtomicReference|2|368952|
> |original|4|555986|new with synchronized|4|540642|new with
> AtomicReference|4|545715|
> |original|8|854029|new with synchronized|8|852137|new with
> AtomicReference|8|850401|
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16476) Remove HLogKey
[ https://issues.apache.org/jira/browse/HBASE-16476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-16476. --- Resolution: Duplicate Resolve as duplicated with HBASE-17132. > Remove HLogKey > -- > > Key: HBASE-16476 > URL: https://issues.apache.org/jira/browse/HBASE-16476 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0 > > Attachments: hbase-16476_v1.patch > > > Deprecated in 1.0+, we should remove HLogKey in 2.0. Cleanup for > coprocessors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16475) Remove SequenceFile based WAL
[ https://issues.apache.org/jira/browse/HBASE-16475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16475: -- Resolution: Duplicate Status: Resolved (was: Patch Available) Resolve as duplicated with HBASE-17132. > Remove SequenceFile based WAL > - > > Key: HBASE-16475 > URL: https://issues.apache.org/jira/browse/HBASE-16475 > Project: HBase > Issue Type: Sub-task > Components: wal >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0 > > Attachments: hbase-16475_v1.patch > > > SequenceFile based WAL is not used in 0.96+. There is no 0.94 -> 2.0 upgrade > (some other upgrade code is removed already), so we can remove SF based WAL > and related code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17132) Cleanup deprecated code for WAL
[ https://issues.apache.org/jira/browse/HBASE-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17132: -- Issue Type: Sub-task (was: Task) Parent: HBASE-16473 > Cleanup deprecated code for WAL > --- > > Key: HBASE-17132 > URL: https://issues.apache.org/jira/browse/HBASE-17132 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Labels: cleanup > Fix For: 2.0.0 > > Attachments: HBASE-17132-v1.patch, HBASE-17132.patch > > > There are some WAL related code which are marked as deprecated since > branch-1(For example the SequenceFileLogWriter). Let's remove it in 2.0 to > keep the code clean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688628#comment-15688628
]
Duo Zhang commented on HBASE-17125:
---
Ping [~giacomotaylor].
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
> Attachments: example.diff
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17132) Cleanup deprecated code for WAL
[ https://issues.apache.org/jira/browse/HBASE-17132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688593#comment-15688593 ] Duo Zhang commented on HBASE-17132: --- Theoretically it is changed but actually it is not for now. The old logic ignore the BaseCoordinatedStateManager and always use zk which is wrong I think. But the only BaseCoordinatedStateManager is zk based for now so they are the same in fact. > Cleanup deprecated code for WAL > --- > > Key: HBASE-17132 > URL: https://issues.apache.org/jira/browse/HBASE-17132 > Project: HBase > Issue Type: Task > Components: wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Labels: cleanup > Fix For: 2.0.0 > > Attachments: HBASE-17132-v1.patch, HBASE-17132.patch > > > There are some WAL related code which are marked as deprecated since > branch-1(For example the SequenceFileLogWriter). Let's remove it in 2.0 to > keep the code clean. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688596#comment-15688596
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
23s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 6s
{color} | {color:red} hbase-assembly in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 9m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 10s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
46m 42s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 13s
{color} | {color:red} root generated 55 new + 18 unchanged - 1 fixed = 73 total
(was 19) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 10s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 10s
{color} | {color:red} hbase-spark-1.6 generated 18 new + 0 unchanged - 0 fixed
= 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 11s
{color} | {color:red} hbase-spark-1.6-scala-2.10 generated 18 new + 0 unchanged
- 0 fixed = 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 11s
{color} | {color:red} hbase-spark-scala-2.10 generated 18 new + 0 unchanged - 0
fixed = 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 1m 0s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 27s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hbase-assembly in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 123m 6s
{color} | {color:green} root in the patch
[jira] [Commented] (HBASE-9774) Provide a way for coprocessors to register and report custom metrics
[ https://issues.apache.org/jira/browse/HBASE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688574#comment-15688574 ] Enis Soztutar commented on HBASE-9774: -- I created an RB for the wip patch depending on our awesome [~elserj]'s work in Apache Calcite. It is not final, but contains almost all of the functionality that we want: - hbase-metrics-api is an HBase-only API that is used internally within HBase, as well as exposed to the coprocessors. This module contains only the API part. The API is very similar to the Dropwizard metrics API and is a subset, with some additions for HBase. Some code is forked from https://github.com/apache/calcite/tree/master/avatica/metrics/src/main/java/org/apache/calcite/avatica/metrics. - hbase-metrics is an implementation of the APIs in the metrics-api module which delegates everything to Dropwizard metrics (3.1.2) implementation. There is also some grouping of metrics in different registries that allows to group metrics into subcomponents (WAL, IPC, etc). This layer does not know anything about Hadoop metrics2. - hadoop2-compat has the Dropwizard2HadoopMetricsAdapter code that knows about both Dropwizard based metrics and Hadoop metrics2. It can collect the metrics in DW registries using metrics2 collectors. This is the only class that is exposed to both of the implementations. Some code is forked from https://github.com/joshelser/dropwizard-hadoop-metrics2. Things that are not coming from calcite: - MetricRegistries / GlobalMetricRegistriesSource: These are for doing JVM-global MetricRegistry instances so that different metric registries can be used for different subcomponents. The global source will collect all of the metrics in all of the DW metric registries. This makes it so that we do not have to define a lot of classes and indirection inside the actual code that uses metrics (hbase-server, etc). - code in hbase-server, etc will mostly use the code from hbase-metrics-api, and do not need to know about dropwizard API and hbase-metrics implementation. - New metric sources (like coprocessors, etc) can use only the new API, and never have to know anything about metrics2. - Even for existing MetricSources (like MetricsRegionServerSource), we can add new metrics using the new API. Existing metrics can be moved to the new API incrementally. The example bulkLoad metrics in MetricsRegionServerSource shows how this can be used. - The patch adds coprocessor metrics at the RS level as well as region level. Some example code shows how to use it from coprocessors. Review is here: https://reviews.apache.org/r/54008/ > Provide a way for coprocessors to register and report custom metrics > > > Key: HBASE-9774 > URL: https://issues.apache.org/jira/browse/HBASE-9774 > Project: HBase > Issue Type: New Feature > Components: Coprocessors, metrics >Reporter: Gary Helmling > > It would help provide better visibility into what coprocessors are doing if > we provided a way for coprocessors to export their own metrics. The general > idea is to: > * extend access to the HBase "metrics bus" down into the coprocessor > environments > * coprocessors can then register and increment custom metrics > * coprocessor metrics are then reported along with all others through normal > mechanisms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16996) Implement storage/retrieval of filesystem-use quotas into quota table
[
https://issues.apache.org/jira/browse/HBASE-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688572#comment-15688572
]
Hadoop QA commented on HBASE-16996:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
27s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
8s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} HBASE-16961 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} HBASE-16961 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 8s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 19s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 31s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 42s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 52s
{color} | {color:red} The patch causes 270 errors with Hadoop v2.5.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 170m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestAvoidCellReferencesIntoShippedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840144/HBASE-16996-HBASE-16961.002.patch
|
| JIRA Issue | HBASE-16996 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Status: Patch Available (was: Open) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 1.2.3, 2.0.0 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4 > > Attachments: HBASE-17165.branch-1.2.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Attachment: HBASE-17165.branch-1.2.001.patch > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4 > > Attachments: HBASE-17165.branch-1.2.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[
https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688525#comment-15688525
]
Hadoop QA commented on HBASE-17165:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 2s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s
{color} | {color:red} hbase-server-jdk1.8.0_111 with JDK v1.8.0_111 generated 3
new + 3 unchanged - 0 fixed = 6 total (was 3) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 30s
{color} | {color:red} hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 3
new + 3 unchanged - 0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 73m 14s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 103m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:d6626eb |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840157/HBASE-17165.branch-1.2.001.patch
|
| JIRA Issue | HBASE-17165 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux de948192a3c7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision |
[jira] [Updated] (HBASE-17146) Reconsider rpc timeout calculation in backup restore operation
[ https://issues.apache.org/jira/browse/HBASE-17146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17146: -- Attachment: HBASE-17146-v1.patch v1. Removed custom hbase.rpc.timeout calculation. Default value is OK. In case of RPC timeout will ever happen, user can increase it in a local hbase-site.xml. cc: [~enis] > Reconsider rpc timeout calculation in backup restore operation > --- > > Key: HBASE-17146 > URL: https://issues.apache.org/jira/browse/HBASE-17146 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-17146-v1.patch > > > We calculate rpc timeout by multiplying # of regions by single_file_timeout > (60 sec) > For big tables this may become very large timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Status: Open (was: Patch Available) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 1.2.3, 2.0.0 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4 > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Attachment: (was: HBASE-17165.branch-1.2.001.patch) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4 > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17166) ITBLL fails on master unable to find hbase-protocol-shaded content
[
https://issues.apache.org/jira/browse/HBASE-17166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17166:
--
Attachment: HBASE-17166.master.001.patch
> ITBLL fails on master unable to find hbase-protocol-shaded content
> --
>
> Key: HBASE-17166
> URL: https://issues.apache.org/jira/browse/HBASE-17166
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: stack
> Attachments: HBASE-17166.master.001.patch
>
>
> From an internal rig found by [~jmhsieh] running generator step:
> {code}
> 16/11/22 16:51:17 INFO mapreduce.Job: Task Id :
> attempt_1479833370377_0002_m_00_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:425)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17166) ITBLL fails on master unable to find hbase-protocol-shaded content
[
https://issues.apache.org/jira/browse/HBASE-17166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17166:
--
Assignee: stack
Affects Version/s: 2.0.0
Status: Patch Available (was: Open)
> ITBLL fails on master unable to find hbase-protocol-shaded content
> --
>
> Key: HBASE-17166
> URL: https://issues.apache.org/jira/browse/HBASE-17166
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
> Attachments: HBASE-17166.master.001.patch
>
>
> From an internal rig found by [~jmhsieh] running generator step:
> {code}
> 16/11/22 16:51:17 INFO mapreduce.Job: Task Id :
> attempt_1479833370377_0002_m_00_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:425)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688493#comment-15688493
]
stack commented on HBASE-17072:
---
Unpatched did 18 seeks. Patched did 17.
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688488#comment-15688488
]
stack commented on HBASE-17072:
---
I ran 5x concurrent scans against my 10M table as HBASE-10676 did. There is no
difference in elapsed time.
Here is one of the scans unpatched:
{code}
Took 971.3250 seconds
real16m16.778s
user7m44.032s
sys 3m16.843s
{code}
... then random patched scan completed in this time:
{code}
...
Took 976.7010 seconds
real16m21.920s
user8m1.396s
sys 3m22.395s
{code}
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17166) ITBLL fails on master unable to find hbase-protocol-shaded content
stack created HBASE-17166:
-
Summary: ITBLL fails on master unable to find
hbase-protocol-shaded content
Key: HBASE-17166
URL: https://issues.apache.org/jira/browse/HBASE-17166
Project: HBase
Issue Type: Bug
Reporter: stack
>From an internal rig found by [~jmhsieh] running generator step:
{code}
16/11/22 16:51:17 INFO mapreduce.Job: Task Id :
attempt_1479833370377_0002_m_00_0, Status : FAILED
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:425)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1790)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688425#comment-15688425
]
Hadoop QA commented on HBASE-17072:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestBlocksScanned |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840153/HBASE-17072.master.002.patch
|
| JIRA Issue | HBASE-17072 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4345862e241a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2b79ac |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4590/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4590/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4590/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4590/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> CPU usage
[jira] [Comment Edited] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686240#comment-15686240
]
Xiang Li edited comment on HBASE-14882 at 11/23/16 12:29 AM:
-
Hi [~anoop.hbase]
When implementing ExtendedCell interface for IndividualBytesFieldCell, to
override the following function
{code}
/**
* Write the given Cell into the given buf's offset.
* @param buf The buffer where to write the Cell.
* @param offset The offset within buffer, to write the Cell.
*/
void write(byte[] buf, int offset);
{code}
I found KeyValue#createByteArray() (as follow) can be re-used
{code}
static byte [] createByteArray(final byte [] row, final int roffset,
final int rlength, final byte [] family, final int foffset, int flength,
final byte [] qualifier, final int qoffset, int qlength,
final long timestamp, final Type type,
final byte [] value, final int voffset,
int vlength, byte[] tags, int tagsOffset, int tagsLength) {
{code}
So I need to check with you
1. If ExtendedCell#write() (listed above) needs to write the Cell in KeyValue
format?
2. If the answer for 1 is yes, then may I propose to make
KeyValue#createByteArray() to package-private so that its logic can be used in
other ExtendedCell implementations?
was (Author: water):
Hi [~anoop.hbase]
When implementing ExtendedCell interface for IndividualBytesFieldCell, to
override the following function
{code}
/**
* Write the given Cell into the given buf's offset.
* @param buf The buffer where to write the Cell.
* @param offset The offset within buffer, to write the Cell.
*/
void write(byte[] buf, int offset);
{code}
I found KeyValue#createByteArray() (as follow) can be re-used
{code}
static byte [] createByteArray(final byte [] row, final int roffset,
final int rlength, final byte [] family, final int foffset, int flength,
final byte [] qualifier, final int qoffset, int qlength,
final long timestamp, final Type type,
final byte [] value, final int voffset,
int vlength, byte[] tags, int tagsOffset, int tagsLength) {
{code}
So I need to check with you
1. If ExtendedCell#write() (listed above) needs to write the Cell in KeyValue
format,
2. If the answer for 1 is yes, then may I propose to make
KeyValue#createByteArray() to package-private so that its logic can be used in
other ExtendedCell implementations?
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch,
> HBASE-14882.master.003.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688343#comment-15688343 ] Hudson commented on HBASE-17160: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2001 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2001/]) HBASE-17160 Undo unnecessary inter-module dependency; spark to hbase-it (stack: rev b2b79ac7d687818c6ea27ac44565d51ae7230d97) * (edit) hbase-it/pom.xml * (edit) hbase-spark/pom.xml > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, hbase.png, minor_hbase.png, untangled_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688315#comment-15688315 ] Ted Yu commented on HBASE-16179: Failed tests are not related to spark module(s). > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, > 16179.v15.txt, 16179.v16.txt, 16179.v18.txt, 16179.v4.txt, 16179.v5.txt, > 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688313#comment-15688313
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m 37s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 59s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
37s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 8m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 12s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
47m 48s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 27s
{color} | {color:red} root generated 55 new + 18 unchanged - 1 fixed = 73 total
(was 19) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hbase-spark-1.6 generated 18 new + 0 unchanged - 0 fixed
= 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hbase-spark-1.6-scala-2.10 generated 18 new + 0 unchanged
- 0 fixed = 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s
{color} | {color:red} hbase-spark-scala-2.10 generated 18 new + 0 unchanged - 0
fixed = 18 total (was 0) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 1m 4s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 26s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 34s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s
{color} | {color:green} hbase-assembly in the patch passed. {color} |
|
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Status: Open (was: Patch Available) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 1.2.3, 2.0.0 >Reporter: Mike Grimes >Priority: Minor > Fix For: 2.0.., 1.2.4 > > Attachments: HBASE-17165.branch-1.2.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Fix Version/s: (was: 2.0..) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4 > > Attachments: HBASE-17165.branch-1.2.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Status: Patch Available (was: Open) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 1.2.3, 2.0.0 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4 > > Attachments: HBASE-17165.branch-1.2.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Attachment: (was: HBASE-17165.master.001.patch) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4, 2.0.. > > Attachments: HBASE-17165.branch-1.2.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Attachment: HBASE-17165.branch-1.2.001.patch > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4, 2.0.. > > Attachments: HBASE-17165.branch-1.2.001.patch, > HBASE-17165.master.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688288#comment-15688288
]
Hadoop QA commented on HBASE-10676:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-10676 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12633590/HBASE-10676-0.98-branchV2.patch
|
| JIRA Issue | HBASE-10676 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4588/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
> perforamce of scan
>
>
> Key: HBASE-10676
> URL: https://issues.apache.org/jira/browse/HBASE-10676
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.99.0
>Reporter: zhaojianbo
>Assignee: zhaojianbo
> Attachments: HBASE-10676-0.98-branch-AtomicReferenceV2.patch,
> HBASE-10676-0.98-branchV2.patch
>
>
> PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
> backward seek operation as the comment said:
> {quote}
> we will not incur a backward seek operation if we have already read this
> block's header as part of the previous read's look-ahead. And we also want to
> skip reading the header again if it has already been read.
> {quote}
> But that is not the case. In the code of 0.98, prefetchedHeader is
> threadlocal for one storefile reader, and in the RegionScanner
> lifecycle，different rpc handlers will serve scan requests of the same
> scanner. Even though one handler of previous scan call prefetched the next
> block header, the other handlers of current scan call will still trigger a
> backward seek operation. The process is like this:
> # rs handler1 serves the scan call, reads block1 and prefetches the header of
> block2
> # rs handler2 serves the same scanner's next scan call, because rs handler2
> doesn't know the header of block2 already prefetched by rs handler1, triggers
> a backward seek and reads block2, and prefetches the header of block3.
> It is not the sequential read. So I think that the threadlocal is useless,
> and should be abandoned. I did the work, and evaluated the performance of one
> client, two client and four client scanning the same region with one
> storefile. The test environment is
> # A hdfs cluster with a namenode, a secondary namenode , a datanode in a
> machine
> # A hbase cluster with a zk, a master, a regionserver in the same machine
> # clients are also in the same machine.
> So all the data is local. The storefile is about 22.7GB from our online data,
> 18995949 kvs. Caching is set 1000. And setCacheBlocks(false)
> With the improvement, the client total scan time decreases 21% for the one
> client case, 11% for the two clients case. But the four clients case is
> almost the same. The details tests' data is the following:
> ||case||client||time(ms)||
> | original | 1 | 306222 |
> | new | 1 | 241313 |
> | original | 2 | 416390 |
> | new | 2 | 369064 |
> | original | 4 | 555986 |
> | new | 4 | 562152 |
> With some modification(see the comments below), the newest result is
> ||case||client||time(ms)||case||client||time(ms)||case||client||time(ms)||
> |original|1|306222|new with synchronized|1|239510|new with
> AtomicReference|1|241243|
> |original|2|416390|new with synchronized|2|365367|new with
> AtomicReference|2|368952|
> |original|4|555986|new with synchronized|4|540642|new with
> AtomicReference|4|545715|
> |original|8|854029|new with synchronized|8|852137|new with
> AtomicReference|8|850401|
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Status: Patch Available (was: Open) > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 1.2.3, 2.0.0 >Reporter: Mike Grimes >Priority: Minor > Fix For: 2.0.., 1.2.4 > > Attachments: HBASE-17165.master.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
[ https://issues.apache.org/jira/browse/HBASE-17165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Grimes updated HBASE-17165: Attachment: HBASE-17165.master.001.patch > Add retry to LoadIncrementalHFiles tool > --- > > Key: HBASE-17165 > URL: https://issues.apache.org/jira/browse/HBASE-17165 > Project: HBase > Issue Type: Improvement > Components: hbase, HFile, tooling >Affects Versions: 2.0.0, 1.2.3 >Reporter: Mike Grimes >Priority: Minor > Fix For: 1.2.4, 2.0.. > > Attachments: HBASE-17165.master.001.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to > failing due to FileNotFoundExceptions due to inconsistency, simple, > configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-17072:
--
Attachment: HBASE-17072.master.002.patch
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688272#comment-15688272
]
stack commented on HBASE-17072:
---
Pushed another patch. It is just additional comments. Can I get a +1 please?
> CPU usage starts to climb up to 90-100% when using G1GC
> ---
>
> Key: HBASE-17072
> URL: https://issues.apache.org/jira/browse/HBASE-17072
> Project: HBase
> Issue Type: Bug
> Components: Performance, regionserver
>Affects Versions: 1.0.0, 2.0.0, 1.2.0
>Reporter: Eiichi Sato
> Attachments: HBASE-17072.master.001.patch,
> HBASE-17072.master.002.patch, disable-block-header-cache.patch,
> mat-threadlocals.png, mat-threads.png, metrics.png, slave1.svg, slave2.svg,
> slave3.svg, slave4.svg
>
>
> h5. Problem
> CPU usage of a region server in our CDH 5.4.5 cluster, at some point, starts
> to gradually get higher up to nearly 90-100% when using G1GC. We've also run
> into this problem on CDH 5.7.3 and CDH 5.8.2.
> In our production cluster, it normally takes a few weeks for this to happen
> after restarting a RS. We reproduced this on our test cluster and attached
> the results. Please note that, to make it easy to reproduce, we did some
> "anti-tuning" on a table when running tests.
> In metrics.png, soon after we started running some workloads against a test
> cluster (CDH 5.8.2) at about 7 p.m. CPU usage of the two RSs started to rise.
> Flame Graphs (slave1.svg to slave4.svg) are generated from jstack dumps of
> each RS process around 10:30 a.m. the next day.
> After investigating heapdumps from another occurrence on a test cluster
> running CDH 5.7.3, we found that the ThreadLocalMap contain a lot of
> contiguous entries of {{HFileBlock$PrefetchedHeader}} probably due to primary
> clustering. This caused more loops in
> {{ThreadLocalMap#expungeStaleEntries()}}, consuming a certain amount of CPU
> time. What is worse is that the method is called from RPC metrics code,
> which means even a small amount of per-RPC time soon adds up to a huge amount
> of CPU time.
> This is very similar to the issue in HBASE-16616, but we have many
> {{HFileBlock$PrefetchedHeader}} not only {{Counter$IndexHolder}} instances.
> Here are some OQL counts from Eclipse Memory Analyzer (MAT). This shows a
> number of ThreadLocal instances in the ThreadLocalMap of a single handler
> thread.
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name =
> "org.apache.hadoop.hbase.io.hfile.HFileBlock$PrefetchedHeader"
> #=> 10980 instances
> {code}
> {code}
> SELECT *
> FROM OBJECTS (SELECT AS RETAINED SET OBJECTS value
> FROM OBJECTS 0x4ee380430) obj
> WHERE obj.@clazz.@name = "org.apache.hadoop.hbase.util.Counter$IndexHolder"
> #=> 2052 instances
> {code}
> Although as described in HBASE-16616 this somewhat seems to be an issue in
> G1GC side regarding weakly-reachable objects, we should keep ThreadLocal
> usage minimal and avoid creating an indefinite number (in this case, a number
> of HFiles) of ThreadLocal instances.
> HBASE-16146 removes ThreadLocals from the RPC metrics code. That may solve
> the issue (I just saw the patch, never tested it at all), but the
> {{HFileBlock$PrefetchedHeader}} are still there in the ThreadLocalMap, which
> may cause issues in the future again.
> h5. Our Solution
> We simply removed the whole {{HFileBlock$PrefetchedHeader}} caching and
> fortunately we didn't notice any performance degradation for our production
> workloads.
> Because the PrefetchedHeader caching uses ThreadLocal and because RPCs are
> handled randomly in any of the handlers, small Get or small Scan RPCs do not
> benefit from the caching (See HBASE-10676 and HBASE-11402 for the details).
> Probably, we need to see how well reads are saved by the caching for large
> Scan or Get RPCs and especially for compactions if we really remove the
> caching. It's probably better if we can remove ThreadLocals without breaking
> the current caching behavior.
> FWIW, I'm attaching the patch we applied. It's for CDH 5.4.5.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17165) Add retry to LoadIncrementalHFiles tool
Mike Grimes created HBASE-17165: --- Summary: Add retry to LoadIncrementalHFiles tool Key: HBASE-17165 URL: https://issues.apache.org/jira/browse/HBASE-17165 Project: HBase Issue Type: Improvement Components: hbase, HFile, tooling Affects Versions: 1.2.3, 2.0.0 Reporter: Mike Grimes Priority: Minor Fix For: 2.0.., 1.2.4 As using the LoadIncrementalHFiles tool with S3 as the filesystem is prone to failing due to FileNotFoundExceptions due to inconsistency, simple, configurable retry logic was added. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688251#comment-15688251
]
Vladimir Rodionov commented on HBASE-17136:
---
{quote}
Are the changes in import necessary ?
{quote}
That *git diff* stuff. I do not add any import statements. Have no idea why git
adds these. I tried creating patch second time
and git again added import lines
{quote}
The list could be quite long. Previously the details were not shown. You don't
need to display every pair, either.
{quote}
No, when you print List - you print all elements of a list
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Minor
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688237#comment-15688237
]
Ted Yu commented on HBASE-17136:
{code}
991 for(Pair pair: list) {
{code}
The list could be quite long. Previously the details were not shown. You don't
need to display every pair, either.
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17136:
---
Priority: Minor (was: Major)
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Minor
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688226#comment-15688226
]
Ted Yu commented on HBASE-17136:
Are the changes in import necessary ?
Showing a sample output with the new format would be nice.
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688207#comment-15688207
]
Hadoop QA commented on HBASE-17136:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-17136 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840146/HBASE-17136-v1.patch |
| JIRA Issue | HBASE-17136 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4586/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v18.txt > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, > 16179.v15.txt, 16179.v16.txt, 16179.v18.txt, 16179.v4.txt, 16179.v5.txt, > 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: (was: 16179.v17.txt) > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, > 16179.v15.txt, 16179.v16.txt, 16179.v18.txt, 16179.v4.txt, 16179.v5.txt, > 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-17156) Restore usage command output formatting is off
[
https://issues.apache.org/jira/browse/HBASE-17156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov resolved HBASE-17156.
---
Resolution: Fixed
Resolved in HBASE-17134
> Restore usage command output formatting is off
> --
>
> Key: HBASE-17156
> URL: https://issues.apache.org/jira/browse/HBASE-17156
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Minor
>
> Formatting is off here:
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase/ restore
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/stack/hbase-2.0.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2016-11-15 22:40:09,081 DEBUG [main] backup.RestoreDriver: Will automatically
> restore all the dependencies
> Usage: bin/hbase restore [options]
> backup_path Path to a backup destination root
> backup_id Backup image ID to restore table(s)Comma-separated
> list of tables to restore
> {quote}
> Restore like backup needs defaults and example args.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-17136:
--
Status: Patch Available (was: Open)
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17136) Fix logging message in backup restore
[
https://issues.apache.org/jira/browse/HBASE-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-17136:
--
Attachment: HBASE-17136-v1.patch
Patch v1. cc: [~tedyu]
> Fix logging message in backup restore
> -
>
> Key: HBASE-17136
> URL: https://issues.apache.org/jira/browse/HBASE-17136
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17136-v1.patch
>
>
> {quote}
> Want to fix this log message so it doesn't have java object ids in it?
> 2016-11-17 14:11:14,639 DEBUG [LoadIncrementalHFiles-2]
> mapreduce.LoadIncrementalHFiles: Going to connect to server
> region=x_1_restore,999adfe23d3bda876af50397a462f7d8-18028,1479420656653.15bd3f57fb7364c5d738f8bf7be7a32c.,
> hostname=ve0540.halxg.cloudera.com,16020,1479274141369, seqNum=2 for row
> 999adfe23d3bda876af50397a462f7d8-18028 with hfile group [
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/33a9cac5e3944e338005dc87e70f85fb}
> ,
> {[B@68c87fc3,hdfs://ve0524.halxg.cloudera.com:8020/user/stack/hbase-staging/restore/fa3bff72113aee83c50c812c722cd8f4/test_cf/935f048447394f93856e917b8dc34a8f}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16996) Implement storage/retrieval of filesystem-use quotas into quota table
[
https://issues.apache.org/jira/browse/HBASE-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-16996:
---
Attachment: HBASE-16996-HBASE-16961.002.patch
.002 rebased on the {{HBASE-16961}} branch and fixed some formatting issues.
> Implement storage/retrieval of filesystem-use quotas into quota table
> -
>
> Key: HBASE-16996
> URL: https://issues.apache.org/jira/browse/HBASE-16996
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0
>
> Attachments: HBASE-16996-HBASE-16961.002.patch, HBASE-16996.001.patch
>
>
> Provide read/write API for accessing the new filesystem-usage quotas in the
> existing {{hbase:quota}} table.
> Make sure that both the client can read quotas the quotas in the table as
> well as the Master can perform the necessary update/delete actions per the
> quota RPCs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17072) CPU usage starts to climb up to 90-100% when using G1GC
[
https://issues.apache.org/jira/browse/HBASE-17072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688165#comment-15688165
]
stack commented on HBASE-17072:
---
Ok. Looks like I had my test counts wrong way around. I see that the current
master branch does about 1/3rd as many more seeks as the patch does.
Here is the seek that current master does that does not happen with the patch
(I have seeks throwing a RuntimeException in a LOG). It happens 143 times in a
small YCSB run that spans loading and workload c:
{code}
26262 2016-11-22 13:40:37,479 INFO
[regionserver/ve0528.halxg.cloudera.com/10.17.240.22:16020-longCompactions-1479850170216]
hfile.HFileBlock: COUNT=583
26263 java.lang.RuntimeException
26264 at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1710)
26265 at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1564)
26266 at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1484)
26267 at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readAndUpdateNewBlock(HFileReaderImpl.java:1124)
26268 at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:1112)
26269 at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:286)
26270 at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:182)
26271 at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:386)
26272 at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:294)
26273 at
org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:250)
26274 at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:515)
26275 at
org.apache.hadoop.hbase.regionserver.compactions.Compactor$1.createScanner(Compactor.java:241)
26276 at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:300)
26277 at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:66)
26278 at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126)
26279 at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1268)
26280 at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2011)
26281 at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:526)
26282 at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:565)
26283 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
26284 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
26285 at java.lang.Thread.run(Thread.java:745)
{code}
The seek that happens with and without patch is the below one which is done on
initial open of the file post compact or on flush or on open of a region:
{code}
613 2016-11-22 13:29:34,730 INFO [StoreFileOpenerThread-info-1]
hfile.HFileBlock: COUNT=1
614 java.lang.RuntimeException
615 at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1710)
616 at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1564)
617 at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$2.nextBlock(HFileBlock.java:1464)
618 at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$2.nextBlockWithBlockType(HFileBlock.java:1472)
619 at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.(HFileReaderImpl.java:209)
620 at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:506)
621 at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:545)
622 at
org.apache.hadoop.hbase.regionserver.StoreFileReader.(StoreFileReader.java:94)
623 at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:270)
624 at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:418)
625 at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:530)
626 at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:520)
627 at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:646)
628 at
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:113)
629 at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:513)
630 at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:510)
631 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
632 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
633 at
[jira] [Updated] (HBASE-17134) Backup command line tool: unification of argument names
[
https://issues.apache.org/jira/browse/HBASE-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17134:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the patch, Vlad.
> Backup command line tool: unification of argument names
> ---
>
> Key: HBASE-17134
> URL: https://issues.apache.org/jira/browse/HBASE-17134
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17134-v1.patch, HBASE-17134-v2.patch
>
>
> {quote}
> So, in backup usage, the backup root dir is called backup_root. In restore,
> it is called backup_path. I presume these are meant to refer to the same
> thing?
> In restore, the id for the backup is the backup_id but in describe when I am
> to pass a backup id, it is called backupId. Ditto on delete. In the dump of
> the history, the backup id is called 'ID'.
> Suggest you use same name for the arg everywhere. The inconsistency makes the
> tool appear untrustworthy.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688153#comment-15688153
]
Hadoop QA commented on HBASE-16993:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 7s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-prefix-tree in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 40s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840124/HBASE-16993.master.004.patch
|
| JIRA Issue | HBASE-16993 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux 51fddb471b24 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2b79ac |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4583/testReport/ |
| modules | C: hbase-prefix-tree hbase-shell hbase-endpoint hbase-it
hbase-spark U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4583/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> BucketCache throw java.io.IOException:
[jira] [Updated] (HBASE-17134) Backup command line tool: unification of argument names
[
https://issues.apache.org/jira/browse/HBASE-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17134:
---
Summary: Backup command line tool: unification of argument names (was:
Command line tool: unification of argument names)
> Backup command line tool: unification of argument names
> ---
>
> Key: HBASE-17134
> URL: https://issues.apache.org/jira/browse/HBASE-17134
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17134-v1.patch, HBASE-17134-v2.patch
>
>
> {quote}
> So, in backup usage, the backup root dir is called backup_root. In restore,
> it is called backup_path. I presume these are meant to refer to the same
> thing?
> In restore, the id for the backup is the backup_id but in describe when I am
> to pass a backup id, it is called backupId. Ditto on delete. In the dump of
> the history, the backup id is called 'ID'.
> Suggest you use same name for the arg everywhere. The inconsistency makes the
> tool appear untrustworthy.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17134) Command line tool: unification of argument names
[
https://issues.apache.org/jira/browse/HBASE-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-17134:
--
Attachment: HBASE-17134-v2.patch
v2 patch.
> Command line tool: unification of argument names
>
>
> Key: HBASE-17134
> URL: https://issues.apache.org/jira/browse/HBASE-17134
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-17134-v1.patch, HBASE-17134-v2.patch
>
>
> {quote}
> So, in backup usage, the backup root dir is called backup_root. In restore,
> it is called backup_path. I presume these are meant to refer to the same
> thing?
> In restore, the id for the backup is the backup_id but in describe when I am
> to pass a backup id, it is called backupId. Ditto on delete. In the dump of
> the history, the backup id is called 'ID'.
> Suggest you use same name for the arg everywhere. The inconsistency makes the
> tool appear untrustworthy.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688128#comment-15688128
]
Dave Latham commented on HBASE-16894:
-
Deriving split points from the actual distribution of data as captured in
HFiles sounds great. However, as a user of a large cluster I'm worried that a
single threaded process doing HDFS operations for every region in a table with
10k, 100k, or more regions could take a great deal of time and/or bottleneck on
the NameNode. Since the region servers already likely have the index data
cached, taking advantage of it would be great if possible.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
> Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v17.txt > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, > 16179.v15.txt, 16179.v16.txt, 16179.v17.txt, 16179.v4.txt, 16179.v5.txt, > 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688051#comment-15688051 ] Sean Busbey commented on HBASE-16179: - can we put the spark release line in the module name consistently? presumably spark will release another incompatible version within the lifetime of what will be hbase 2.y, and I'd rather not have the meaning of the undecorated module change. (I see that it's consistent in the artifact) > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, > 16179.v15.txt, 16179.v16.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, > 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17160: -- Attachment: untangled_hbase.png Picture of what stuff is like after addendum goes in. TODO is visibility labels and ByteStringer. These require module dependencies we might be able to undo. Now our hierarchy is flatter. Up top is hbase-examples which depends on loads of stuff. Then next tier is client stuff (thrift, rest, endpoints, spark), then at the heart, hbase-server. Below this are client, compat, procedure, prefix and then hbase-protocol* and hbase-common. At the base are the annotations that all use. This ain't too bad. Will work on the rest in new issue. > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, hbase.png, minor_hbase.png, untangled_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688027#comment-15688027 ] stack commented on HBASE-17160: --- Pushed an update on the addendum. Lets see how it does. > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, hbase.png, minor_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16904) remove directory layout / fs references from snapshots
[
https://issues.apache.org/jira/browse/HBASE-16904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687993#comment-15687993
]
Hadoop QA commented on HBASE-16904:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s {color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 38s
{color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s {color} | {color:green} hbase-14439 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
59s {color} | {color:green} hbase-14439 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 31s
{color} | {color:red} hbase-server in hbase-14439 has 5 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s
{color} | {color:green} hbase-14439 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m 16s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 53s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 59s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 58s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 0s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 0s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 58s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 55s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 52s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 49s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 46s
{color} | {color:red} The patch causes 215 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue}
[jira] [Updated] (HBASE-16993) BucketCache throw java.io.IOException: Invalid HFile block magic when DATA_BLOCK_ENCODING set to DIFF
[
https://issues.apache.org/jira/browse/HBASE-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16993:
--
Attachment: HBASE-16993.master.004.patch
> BucketCache throw java.io.IOException: Invalid HFile block magic when
> DATA_BLOCK_ENCODING set to DIFF
> -
>
> Key: HBASE-16993
> URL: https://issues.apache.org/jira/browse/HBASE-16993
> Project: HBase
> Issue Type: Bug
> Components: BucketCache, io
>Affects Versions: 1.1.3
> Environment: hbase version 1.1.3
>Reporter: liubangchen
>Assignee: liubangchen
> Fix For: 2.0.0
>
> Attachments: HBASE-16993.000.patch, HBASE-16993.001.patch,
> HBASE-16993.master.001.patch, HBASE-16993.master.002.patch,
> HBASE-16993.master.003.patch, HBASE-16993.master.004.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> hbase-site.xml setting
>
> hbase.bucketcache.bucket.sizes
> 16384,32768,40960,
> 46000,49152,51200,65536,131072,524288
>
>
> hbase.bucketcache.size
> 16384
>
>
> hbase.bucketcache.ioengine
> offheap
>
>
> hfile.block.cache.size
> 0.3
>
>
> hfile.block.bloom.cacheonwrite
> true
>
>
> hbase.rs.cacheblocksonwrite
> true
>
>
> hfile.block.index.cacheonwrite
> true
> n_splits = 200
> create 'usertable',{NAME =>'family', COMPRESSION => 'snappy', VERSIONS =>
> 1,DATA_BLOCK_ENCODING => 'DIFF',CONFIGURATION =>
> {'hbase.hregion.memstore.block.multiplier' => 5}},{DURABILITY =>
> 'SKIP_WAL'},{SPLITS => (1..n_splits).map {|i|
> "user#{1000+i*(-1000)/n_splits}"}}
> load data
> bin/ycsb load hbase10 -P workloads/workloada -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> recordcount=2 -p insertorder=hashed -p insertstart=0 -p
> clientbuffering=true -p durability=SKIP_WAL -threads 20 -s
> run
> bin/ycsb run hbase10 -P workloads/workloadb -p table=usertable -p
> columnfamily=family -p fieldcount=10 -p fieldlength=100 -p
> operationcount=2000 -p readallfields=true -p clientbuffering=true -p
> requestdistribution=zipfian -threads 10 -s
> log info
> 2016-11-02 20:20:20,261 ERROR
> [RW.default.readRpcServer.handler=36,queue=21,port=6020] bucket.BucketCache:
> Failed reading block fdcc7ed6f3b2498b9ef316cc8206c233_44819759 from bucket
> cache
> java.io.IOException: Invalid HFile block magic:
> \x00\x00\x00\x00\x00\x00\x00\x00
> at
> org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:154)
> at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:273)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:134)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$1.deserialize(HFileBlock.java:121)
> at
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.getBlock(BucketCache.java:427)
> at
> org.apache.hadoop.hbase.io.hfile.CombinedBlockCache.getBlock(CombinedBlockCache.java:85)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.getCachedBlock(HFileReaderV2.java:266)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:403)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:269)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:634)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:584)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:247)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:156)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2071)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5369)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2546)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2532)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2514)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6558)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6537)
> at
>
