[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105349#comment-15105349
]
Hadoop QA commented on HBASE-15125:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 9s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 35s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 90, now 90). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 149m 54s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 14s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 321m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests |
hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
| JDK v1.8.0 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782832/HBASE-15125-V001.patch
|
| JIRA Issue | HBASE-15125 |
| Optional Tests
[jira] [Updated] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14969:
--
Attachment: HBASE-14969_v10.patch
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105285#comment-15105285
]
Yu Li commented on HBASE-14969:
---
The two findbugs issues don't need to handle as mentioned before.
The hadoopcheck issue is tracked by HBASE-15103, we could see the same error in
log:
{noformat}
[ERROR] Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList
at META-INF/LICENSE.vm[line 1627, column 22]
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:395)
at
org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:384)
at
org.apache.velocity.runtime.parser.node.ASTIndex.execute(ASTIndex.java:149)
{noformat}
Will update the patch to resolve new checkstyle issues.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v2.patch,
> HBASE-14969_v3.patch, HBASE-14969_v4.patch, HBASE-14969_v5.patch,
> HBASE-14969_v6.patch, HBASE-14969_v9.patch, load-nothrottling.log,
> load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105296#comment-15105296
]
Hadoop QA commented on HBASE-15126:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 135m 4s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 123m 12s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 302m 44s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782829/HBASE-15126-v002.patch
|
| JIRA Issue | HBASE-15126 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105316#comment-15105316
]
chenrongwei commented on HBASE-15097:
-
Thanks for your reminder. I think there are other two bugs which had been
hidden by this bug. I also had submit the two bugs just in HBASE-15125 and
HBASE-15126. I think that we have to solve that two bugs first before this.
Additionally,I don't understand what does "hadoopcheck Patch causes 11 errors
with Hadoop v2.4.0." means?
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105258#comment-15105258
]
Ted Yu commented on HBASE-15097:
Can you check whether hbck test failure was related to the patch ?
Please modify the subject of JIRA to closely reflect what the patch does.
Thanks
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenrongwei updated HBASE-15097:
Attachment: HBASE-15097-v002.patch
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105036#comment-15105036
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 22s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 18s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 54s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 21s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 568, now 563). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 18s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 37s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 54s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 13s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 31s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 50s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 10s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 30s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 49s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s
{color} | {color:red}
[jira] [Updated] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[ https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15125: Status: Patch Available (was: Open) > HBaseFsck's adoptHdfsOrphan function create region with wrong end key > boundary. > --- > > Key: HBASE-15125 > URL: https://issues.apache.org/jira/browse/HBASE-15125 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15125-V001.patch > > > There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this > function will create a region,which want to cover all the orphan regions.But > the end key of this new region was set incorrectly.Correct region's boundary > should be [startKey,endKey),but this function create a region with boundary > of [startKey,endKey],this bug will leads to scan operation omit some data. > I think we should create the region like bellow, > // create new region on hdfs. move data into place. > HRegionInfo hri = new HRegionInfo(template.getTableName(), > orphanRegionRange.getFirst(), > Bytes.add(orphanRegionRange.getSecond(), new byte[1])); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[ https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15125: Attachment: HBASE-15125-V001.patch > HBaseFsck's adoptHdfsOrphan function create region with wrong end key > boundary. > --- > > Key: HBASE-15125 > URL: https://issues.apache.org/jira/browse/HBASE-15125 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15125-V001.patch > > > There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this > function will create a region,which want to cover all the orphan regions.But > the end key of this new region was set incorrectly.Correct region's boundary > should be [startKey,endKey),but this function create a region with boundary > of [startKey,endKey],this bug will leads to scan operation omit some data. > I think we should create the region like bellow, > // create new region on hdfs. move data into place. > HRegionInfo hri = new HRegionInfo(template.getTableName(), > orphanRegionRange.getFirst(), > Bytes.add(orphanRegionRange.getSecond(), new byte[1])); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[ https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15126: Attachment: HBASE-15126-v002.patch > HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was > incorrect. > --- > > Key: HBASE-15126 > URL: https://issues.apache.org/jira/browse/HBASE-15126 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch > > > HBaseFsck's checkRegionBoundaries function set the > 'currentRegionBoundariesInformation.storesFirstKey' was incorrect.I think it > should be set like below, > currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey); > but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just > set to 'storeFirstKey',which will cause to > comparator.compare(currentRegionBoundariesInformation.storesFirstKey, > currentRegionBoundariesInformation.metaFirstKey) get a wrong > result. Because it just compared the rowkey's length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10605) Manage the call timeout in the server
[ https://issues.apache.org/jira/browse/HBASE-10605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105088#comment-15105088 ] Nicolas Liochon commented on HBASE-10605: - Hi [~java8964], what do you need to know? The point of this jira if that the server should not continue to handle a request if we know that the client has already stopped waiting for the result. > Manage the call timeout in the server > - > > Key: HBASE-10605 > URL: https://issues.apache.org/jira/browse/HBASE-10605 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.99.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > > Since HBASE-10566, we have an explicit call timeout available in the client. > We could forward it to the server, and use this information for: > - if the call is still in the queue, just cancel it > - if the call is under execution, makes this information available in > RpcCallContext (actually change the RpcCallContext#disconnectSince to > something more generic), so it can be used by the query under execution to > stop its execution > - in the future, interrupt it to manage the case 'stuck on a dead datanode' > or something similar > - if the operation has finished, don't send the reply to the client, as by > definition the client is not interested anymore. > From this, it will be easy to manage the cancellation: > disconnect/timeout/cancellation are similar from a service execution PoV -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10605) Manage the call timeout in the server
[ https://issues.apache.org/jira/browse/HBASE-10605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-10605: Assignee: (was: Nicolas Liochon) > Manage the call timeout in the server > - > > Key: HBASE-10605 > URL: https://issues.apache.org/jira/browse/HBASE-10605 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, regionserver >Affects Versions: 0.99.0 >Reporter: Nicolas Liochon > > Since HBASE-10566, we have an explicit call timeout available in the client. > We could forward it to the server, and use this information for: > - if the call is still in the queue, just cancel it > - if the call is under execution, makes this information available in > RpcCallContext (actually change the RpcCallContext#disconnectSince to > something more generic), so it can be used by the query under execution to > stop its execution > - in the future, interrupt it to manage the case 'stuck on a dead datanode' > or something similar > - if the operation has finished, don't send the reply to the client, as by > definition the client is not interested anymore. > From this, it will be easy to manage the cancellation: > disconnect/timeout/cancellation are similar from a service execution PoV -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15119) Include git SHA in check_compatibility reports
[ https://issues.apache.org/jira/browse/HBASE-15119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105557#comment-15105557 ] Andrew Purtell commented on HBASE-15119: +1 > Include git SHA in check_compatibility reports > -- > > Key: HBASE-15119 > URL: https://issues.apache.org/jira/browse/HBASE-15119 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15119.v00.patch > > > Since some refs change over time (ie, branches), it would be nice to include > git shas in the version info included in check compatibility reports. It'll > also help interested parties to be sure of what they're looking at. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105566#comment-15105566
]
Yu Li commented on HBASE-14969:
---
The latest HadoopQA run looks good.
> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.patch, HBASE-14969_v10.patch,
> HBASE-14969_v2.patch, HBASE-14969_v3.patch, HBASE-14969_v4.patch,
> HBASE-14969_v5.patch, HBASE-14969_v6.patch, HBASE-14969_v9.patch,
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike
> caused by huge IO pressure like network/disk overflow. However, even with
> this control on, we are still observing disk utils near 100%, and by analysis
> we think this is caused by flush, especially when we increase the setting of
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a
> supplement of HBASE-8329 to better control IO pressure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105613#comment-15105613
]
Ted Yu commented on HBASE-15073:
bq. why would I want the 'choice' of turning off splits?
The following description applies to time series data with FIFO compaction
policy enforced (used by Ambari Metrics Server).
If a region continues to receive write requests, ultimately it would split -
without intervention from normalizer.
On the other hand, if the region stops receiving new write request at some
point, its size would stabilize. After TTL passes for all the HFiles in the
region, FIFO compaction would make the region empty.
So there doesn't need to be split request sent from normalizer.
In fact, the split of regions not actively receiving writes increases the work
in the future because there would be merge request(s) when such regions become
empty after compaction policy kicks in.
I talked with an Ambari developer who is involved in development of Ambari
Metrics Server and verified the above observation.
bq. where you turn normalization on and it just does the right thing
autonomously
Letting region normalizer do the work autonomously is good but we're not there
yet.
This finer grained control allows various use cases benefit from region
normalizer given the current implementation.
Again, I am open to better naming of the normalization choices.
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105234#comment-15105234
]
Hadoop QA commented on HBASE-15097:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 9s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 34s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 3s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 38s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 12s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 42s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 15s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 50s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 36s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 14m 20s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with
[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[ https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105488#comment-15105488 ] Ted Yu commented on HBASE-15125: Can you add unit test showing the issue ? Thanks > HBaseFsck's adoptHdfsOrphan function create region with wrong end key > boundary. > --- > > Key: HBASE-15125 > URL: https://issues.apache.org/jira/browse/HBASE-15125 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15125-V001.patch > > > There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this > function will create a region,which want to cover all the orphan regions.But > the end key of this new region was set incorrectly.Correct region's boundary > should be [startKey,endKey),but this function create a region with boundary > of [startKey,endKey],this bug will leads to scan operation omit some data. > I think we should create the region like bellow, > // create new region on hdfs. move data into place. > HRegionInfo hri = new HRegionInfo(template.getTableName(), > orphanRegionRange.getFirst(), > Bytes.add(orphanRegionRange.getSecond(), new byte[1])); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[ https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105493#comment-15105493 ] Ted Yu commented on HBASE-15126: Is it possible to add a test for this issue ? > HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was > incorrect. > --- > > Key: HBASE-15126 > URL: https://issues.apache.org/jira/browse/HBASE-15126 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch > > > HBaseFsck's checkRegionBoundaries function set the > 'currentRegionBoundariesInformation.storesFirstKey' was incorrect.I think it > should be set like below, > currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey); > but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just > set to 'storeFirstKey',which will cause to > comparator.compare(currentRegionBoundariesInformation.storesFirstKey, > currentRegionBoundariesInformation.metaFirstKey) get a wrong > result. Because it just compared the rowkey's length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15121) ConnectionImplementation#locateRegionInMeta() issue when master is restarted
[
https://issues.apache.org/jira/browse/HBASE-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samir Ahmic updated HBASE-15121:
Description:
I notice this issue while i was running IntegrationTestMTTR#testRestartMaster()
test was failing on put operation. Here is sequence of events from logs leading
to failed put operation:
Master restart
{code}
INFO [pool-5-thread-1] util.Shell: Executing full command [/usr/bin/ssh
hnode2 "sudo -u hbase ps aux | grep proc_master | grep -v grep | tr -s ' ' |
cut -d ' ' -f2 | xargs kill -s SIGKILL"]
{code}
Client trying to locate region for row=70efdf2ec9b086079795c442636b55fb-17
(this is additional logging inspecting metaKey which is used to search
hbase:meta )
{code}
2016-01-15 10:26:05,169 INFO [HBaseWriterThread_9]
client.ConnectionImplementation: metaKey inspection:
table=IntegrationTestMTTRLoadTestTool row= 70efdf2ec9b086079795c442636b55fb-17
metaKey=
IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
{code}
Client throwing TableNotFoundException (hbase:meta scan returned null)
{code}
2016-01-15 10:32:58,154 INFO [HBaseWriterThread_5]
client.ConnectionImplementation: regionInfo result is null: HBaseWriterThread_5
throwing TableNotFoundException logging details
table=IntegrationTestMTTRLoadTestTool row=70efdf2ec9b086079795c442636b55fb-17
metaKey=IntegrationTestMTTRLoadTestTool,70efdf2ec9b086079795c442636b55fb-17,99
2016-01-15 10:32:58,154 ERROR [HBaseWriterThread_5] client.AsyncProcess: Failed
to get region location
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
at
org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
at
org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
{code}
And as result we have failed insert operation:
{code}
2016-01-15 10:32:58,179 ERROR [HBaseWriterThread_5] util.MultiThreadedWriter:
Failed to insert: 17 after 60046ms; region information: cached:
region=IntegrationTestMTTRLoadTestTool,6660,1452849956427.05b437185a9437f178726a55a29a79b7.,
hostname=hnode4,16020,1452776418437, seqNum=5; cache is up to date; errors:
exception from null for 70efdf2ec9b086079795c442636b55fb-17
org.apache.hadoop.hbase.TableNotFoundException: IntegrationTestMTTRLoadTestTool
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:890)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:781)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:396)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:239)
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:569)
at
org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.insert(MultiThreadedWriter.java:146)
at
org.apache.hadoop.hbase.util.MultiThreadedWriter$HBaseWriterThread.run(MultiThreadedWriter.java:111)
{code}
leading to test failing:
{code}
Failed to write key: 17
2016-01-15 10:33:53,984 INFO [main] mttr.IntegrationTestMTTR: RestartMaster
failed after 469878ms.
java.util.concurrent.ExecutionException: java.lang.AssertionError: Load failed
expected:<0> but was:<1>
{code}
Here is snippet from ConnectionImplementation#locateRegionInMeta() throwing
exception:
{code}
try {
Result regionInfoRow = null;
ReversedClientScanner rcs = null;
try {
rcs = new ClientSmallReversedScanner(conf, s,
TableName.META_TABLE_NAME, this,
rpcCallerFactory, rpcControllerFactory, getMetaLookupPool(), 0);
regionInfoRow = rcs.next();
} finally {
if (rcs != null) {
rcs.close();
}
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105496#comment-15105496
]
Ted Yu commented on HBASE-15097:
The last test run was done on https://builds.apache.org/computer/H2 where local
maven repo may contain outdated artifacts.
This led to "hadoopcheck Patch causes 11 errors with Hadoop v2.4.0.", seen in
other QA runs before.
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14969) Add throughput controller for flush
[
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105521#comment-15105521
]
Hadoop QA commented on HBASE-14969:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 82 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 16s {color}
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 47s {color}
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 19s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 38s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 57s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 15s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 35s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 55s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 15s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 34s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 54s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hbase-server introduced 2 new FindBugs issues. {color} |
|
[jira] [Updated] (HBASE-15035) bulkloading hfiles with tags that require splits do not preserve tags
[ https://issues.apache.org/jira/browse/HBASE-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15035: -- Fix Version/s: (was: 1.0.4) 1.0.3 > bulkloading hfiles with tags that require splits do not preserve tags > - > > Key: HBASE-15035 > URL: https://issues.apache.org/jira/browse/HBASE-15035 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.98.0, 1.0.0, 2.0.0, 1.1.0, 1.2.0, 1.3.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: HBASE-15035-v2.patch, HBASE-15035-v3.patch, > HBASE-15035-v4.patch, HBASE-15035.patch > > > When an hfile is created with cell tags present and it is bulk loaded into > hbase the tags will be present when loaded into a single region. If the bulk > load hfile spans multiple regions, bulk load automatically splits the > original hfile into a set of split hfiles corresponding to each of the > regions that the original covers. > Since 0.98, tags are not copied into the newly created split hfiles. (the > default for "includeTags" of the HFileContextBuilder [1] is uninitialized > which defaults to false). This means acls, ttls, mob pointers and other tag > stored values will not be bulk loaded in. > [1] > https://github.com/apache/hbase/blob/master/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java#L40 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14845) hbase-server leaks jdk.tools dependency to mapreduce consumers
[ https://issues.apache.org/jira/browse/HBASE-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14845: -- Fix Version/s: (was: 1.0.4) 1.0.3 > hbase-server leaks jdk.tools dependency to mapreduce consumers > -- > > Key: HBASE-14845 > URL: https://issues.apache.org/jira/browse/HBASE-14845 > Project: HBase > Issue Type: Bug > Components: build, dependencies >Affects Versions: 2.0.0, 0.98.14, 1.2.0, 1.1.2, 1.3.0, 1.0.3 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.4, 0.98.18 > > > HBASE-13963 / HBASE-14844 take care of removing leaks of our dependency on > jdk-tools. > Until we move the mapreduce support classes out of hbase-server > (HBASE-11843), we need to also avoid leaking the dependency from that module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14516) categorize hadoop-compat tests
[ https://issues.apache.org/jira/browse/HBASE-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14516: -- Fix Version/s: (was: 1.0.4) 1.0.3 > categorize hadoop-compat tests > -- > > Key: HBASE-14516 > URL: https://issues.apache.org/jira/browse/HBASE-14516 > Project: HBase > Issue Type: Task > Components: build, hadoop2, test >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: HBASE-14516.0.98.v2.patch, HBASE-14516.1.patch > > > the hadoop-compat and hadoop2-compat modules do not rely on the hbase > annotations test-jar and their tests aren't categorized. > this causes things to fail if you attempt to specify one of our test > categories to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15085:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
>Priority: Critical
> Labels: hfile
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-0.98-v5.patch,
> HBASE-15085-branch-1.0-v1.patch, HBASE-15085-branch-1.0-v2.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.1-v2.patch,
> HBASE-15085-branch-1.2-v1.patch, HBASE-15085-branch-1.2-v2.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch, HBASE-15085-v5.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs = inFile.getFileSystem(conf);
> CacheConfig cacheConf = new CacheConfig(conf);
> HalfStoreFileReader halfReader =
[jira] [Updated] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability
[ https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14799: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Commons-collections object deserialization remote command execution > vulnerability > -- > > Key: HBASE-14799 > URL: https://issues.apache.org/jira/browse/HBASE-14799 > Project: HBase > Issue Type: Bug > Components: dependencies, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 2.0.0, 0.94.28, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: 14799-0.98.addendum, HBASE-14799-0.94.patch, > HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, > HBASE-14799-0.94.patch, HBASE-14799-0.98.patch, HBASE-14799-0.98.patch, > HBASE-14799-0.98.patch, HBASE-14799.patch, HBASE-14799.patch > > > Read: > http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/ > TL;DR: If you have commons-collections on your classpath and accept and > process Java object serialization data, then you probably have an exploitable > remote command execution vulnerability. > 0.94 and earlier HBase releases are vulnerable because we might read in and > rehydrate serialized Java objects out of RPC packet data in > HbaseObjectWritable using ObjectInputStream#readObject (see > https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714) > and we have commons-collections on the classpath on the server. > 0.98 also carries some limited exposure to this problem through inclusion of > backwards compatible deserialization code in > HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration > utility, and by the AccessController when reading permissions from the ACL > table serialized in legacy format by 0.94. Unprivileged users cannot run the > tool nor access the ACL table. > Unprivileged users can however attack a 0.94 installation. An attacker might > be able to use the method discussed on that blog post to capture valid HBase > RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, > and replay them to trigger a remote command execution with the privileges of > the account under which the HBase RegionServer daemon is running. > We need to make a patch release of 0.94 that changes HbaseObjectWritable to > disallow processing of random Java object serializations. This will be a > compatibility break that might affect old style coprocessors, which quite > possibly may rely on this catch-all in HbaseObjectWritable for custom object > (de)serialization. We can introduce a new configuration setting, > "hbase.allow.legacy.object.serialization", defaulting to false. > To be thorough, we can also use the new configuration setting > "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to > prevent the AccessController from falling back to the vulnerable legacy code. > This turns out to not affect the ability to migrate permissions because > TablePermission implements Writable, which is safe, not Serializable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14839) [branch-1] Backport test categories so that patch backport is easier
[
https://issues.apache.org/jira/browse/HBASE-14839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14839:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> [branch-1] Backport test categories so that patch backport is easier
>
>
> Key: HBASE-14839
> URL: https://issues.apache.org/jira/browse/HBASE-14839
> Project: HBase
> Issue Type: Test
> Components: test
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: hbase-14839-branch-1.patch
>
>
> Test categories are in master and new unit tests are sometimes marked with
> that particular interface ( {{RPCTests.class}} ).
> Since we don't have the specific annotation classes in branch-1, backports
> usually fail. We can just commit those classes to all applicable branches so
> that committing patches is less work.
> We can also backport the full patch for running the specific tests from maven
> as a further issue. Feel free to take it up, if you are interested.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14930) check_compatibility.sh needs smarter exit codes
[ https://issues.apache.org/jira/browse/HBASE-14930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14930: -- Fix Version/s: (was: 1.0.4) 1.0.3 > check_compatibility.sh needs smarter exit codes > --- > > Key: HBASE-14930 > URL: https://issues.apache.org/jira/browse/HBASE-14930 > Project: HBase > Issue Type: Bug >Reporter: Dima Spivak >Assignee: Dima Spivak > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: HBASE-14930_master_v1.patch > > > The check_compatibility.sh tool in dev_support uses the Java API Compliance > Checker to do static analysis of source/binary incompatibilties between two > HBase branches. One problem, though, is that the script has a few instances > where it may return an exit code of 1 (e.g. if Maven steps fail), but this is > the same exit code that the Java ACC tool itself uses to denote that the tool > succeeded, but found incompatibilities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14904) Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
[
https://issues.apache.org/jira/browse/HBASE-14904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14904:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> Mark Base[En|De]coder LimitedPrivate and fix binary compat issue
>
>
> Key: HBASE-14904
> URL: https://issues.apache.org/jira/browse/HBASE-14904
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: hbase-14904_v1.patch, hbase-14904_v2.patch
>
>
> PHOENIX-2477 revealed that the changes from HBASE-14501 breaks binary
> compatibility in Phoenix compiled with earlier versions of HBase and run
> agains later versions.
> This is one of the areas that the boundary is not clear, but it won't hurt us
> to fix it.
> The exception trace is:
> {code}
> Exception in thread "main" java.lang.NoSuchFieldError: in
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$PhoenixBaseDecoder.(IndexedWALEditCodec.java:106)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec$IndexKeyValueDecoder.(IndexedWALEditCodec.java:121)
> at
> org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec.getDecoder(IndexedWALEditCodec.java:63)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initAfterCompression(ProtobufLogReader.java:292)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:82)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:148)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:316)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:281)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:269)
> at
> org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:418)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.processFile(WALPrettyPrinter.java:247)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.run(WALPrettyPrinter.java:422)
> at
> org.apache.hadoop.hbase.wal.WALPrettyPrinter.main(WALPrettyPrinter.java:357)
> {code}
> Although {{BaseDecoder.in}} is still there, it got changed to be a class
> rather than an interface. BaseDecoder is marked Private, thus the binary
> compat check is not run at all. Not sure whether it would have caught this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14989) Implementation of Mutation.getWriteToWAL() is backwards
[
https://issues.apache.org/jira/browse/HBASE-14989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14989:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> Implementation of Mutation.getWriteToWAL() is backwards
> ---
>
> Key: HBASE-14989
> URL: https://issues.apache.org/jira/browse/HBASE-14989
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: James Taylor
>Assignee: Enis Soztutar
> Fix For: 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-14989.1.branch-1.patch,
> hbase-14989-branch-1_v1.patch
>
>
> The implementation of the deprecated getWriteToWAL is backwards. It should
> return true if this.durability == Durability.SYNC_WAL:
> {code}
> /**
>* @deprecated Use {@link #getDurability()} instead.
>* @return true if edits should be applied to WAL, false if not
>*/
> @Deprecated
> public boolean getWriteToWAL() {
> return this.durability == Durability.SKIP_WAL;
> }
> {code}
> For example, if mutation.durability is Durability.SYNC_WAL and the following
> code is called {{clonedMutation.setWriteToWAL(mutation.getWriteToWAL())}}, it
> will disable writing to the WAL for clonedMutation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14936) CombinedBlockCache should overwrite CacheStats#rollMetricsPeriod()
[
https://issues.apache.org/jira/browse/HBASE-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14936:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> CombinedBlockCache should overwrite CacheStats#rollMetricsPeriod()
> --
>
> Key: HBASE-14936
> URL: https://issues.apache.org/jira/browse/HBASE-14936
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.1.2
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3
>
> Attachments: HBASE-14936-branch-1.0-1.1.patch,
> HBASE-14936-branch-1.0-addendum.patch, HBASE-14936-trunk-v1.patch,
> HBASE-14936-trunk-v2.patch, HBASE-14936-trunk.patch
>
>
> It seems CombinedBlockCache should overwrite CacheStats#rollMetricsPeriod() as
> {code}
> public void rollMetricsPeriod() {
> lruCacheStats.rollMetricsPeriod();
> bucketCacheStats.rollMetricsPeriod();
> }
> {code}
> otherwise, CombinedBlockCache.getHitRatioPastNPeriods() and
> CombinedBlockCache.getHitCachingRatioPastNPeriods() will always return 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14968) ConcurrentModificationException in region close resulting in the region staying in closing state
[
https://issues.apache.org/jira/browse/HBASE-14968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14968:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> ConcurrentModificationException in region close resulting in the region
> staying in closing state
>
>
> Key: HBASE-14968
> URL: https://issues.apache.org/jira/browse/HBASE-14968
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment, regionserver
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: hbase-14968_v1.patch
>
>
> We have seen this in our tests. The region gets closed, but the region close
> handler gets an unexpected exception causing the region to stay in closing
> state until RS is restarted.
> The Phoenix and security coprocessors were loaded in that region. Here is the
> stack trace:
> {code}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.shutdown(CoprocessorHost.java:442)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment.shutdown(RegionCoprocessorHost.java:155)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.shutdown(CoprocessorHost.java:272)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$5.postEnvCall(RegionCoprocessorHost.java:496)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1761)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postClose(RegionCoprocessorHost.java:489)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1502)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1349)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14761) Deletes with and without visibility expression do not delete the matching mutation
[
https://issues.apache.org/jira/browse/HBASE-14761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14761:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> Deletes with and without visibility expression do not delete the matching
> mutation
> --
>
> Key: HBASE-14761
> URL: https://issues.apache.org/jira/browse/HBASE-14761
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 1.0.0, 1.0.1, 1.1.0, 1.0.2, 1.1.2, 0.98.15
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-14761.patch, HBASE-14761_0.98_addendum.patch
>
>
> This is from the user list as reported by Anoop Sharma
> {code}
> running into an issue related to visibility expressions and delete.
> Example run from hbase shell is listed below.
> Will appreciate any help on this issue.
> thanks.
> In the example below, user running queries has ‘MANAGER’ authorization.
> *First example:*
> add a column with visib expr ‘MANAGER’
> delete it by passing in visibility of ‘MANAGER’
> This works and scan doesn’t return anything.
> *Second example:*
> add a column with visib expr ‘MANAGER’
> delete it by not passing in any visibility.
> This doesn’t delete the column.
> Scan doesn’t return the row but RAW scan shows the column
> marked as deleteColumn.
> Now if delete is done again with visibility of ‘MANAGER’,
> it still doesn’t delete it and scan returns the original column.
> *Example 1:*
> hbase(main):096:0> create 'HBT1', 'cf'
> hbase(main):098:0* *put 'HBT1', 'John', 'cf:a', 'CA',
> {VISIBILITY=>'MANAGER'}*
> hbase(main):099:0> *scan 'HBT1'*
> ROW
> COLUMN+CELL
> John column=cf:a, timestamp=1446154722055,
> value=CA
> 1 row(s) in 0.0030 seconds
> hbase(main):100:0> *delete 'HBT1', 'John', 'cf:a', {VISIBILITY=>'MANAGER'}*
> 0 row(s) in 0.0030 seconds
> hbase(main):101:0> *scan 'HBT1'*
> ROW
> COLUMN+CELL
> 0 row(s) in 0.0030 seconds
> *Example 2:*
> hbase(main):010:0* *put 'HBT1', 'John', 'cf:a', 'CA',
> {VISIBILITY=>'MANAGER'}*
> 0 row(s) in 0.0040 seconds
> hbase(main):011:0> *scan 'HBT1'*
> ROW
> COLUMN+CELL
> John column=cf:a, timestamp=1446155346473,
> value=CA
> 1 row(s) in 0.0060 seconds
> hbase(main):012:0> *delete 'HBT1', 'John', 'cf:a'*
> 0 row(s) in 0.0090 seconds
> hbase(main):013:0> *scan 'HBT1'*
> ROW
> COLUMN+CELL
> John column=cf:a, timestamp=1446155346473,
> value=CA
> 1 row(s) in 0.0050 seconds
> hbase(main):014:0> *scan 'HBT1', {RAW => true}*
> ROW
> COLUMN+CELL
> John column=cf:a, timestamp=1446155346519,
> type=DeleteColumn
> 1 row(s) in 0.0060 seconds
> hbase(main):015:0> *delete 'HBT1', 'John', 'cf:a', {VISIBILITY=>'MANAGER'}*
> 0 row(s) in 0.0030 seconds
> hbase(main):016:0> *scan 'HBT1'*
> ROW
> COLUMN+CELL
> John column=cf:a, timestamp=1446155346473,
> value=CA
> 1 row(s) in 0.0040 seconds
> hbase(main):017:0> *scan 'HBT1', {RAW => true}*
> ROW
> COLUMN+CELL
> John column=cf:a, timestamp=1446155346601,
> type=DeleteColumn
> 1 row(s) in 0.0060 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14806) Missing sources.jar for several modules when building HBase
[
https://issues.apache.org/jira/browse/HBASE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14806:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> Missing sources.jar for several modules when building HBase
> ---
>
> Key: HBASE-14806
> URL: https://issues.apache.org/jira/browse/HBASE-14806
> Project: HBase
> Issue Type: Bug
> Components: pom
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-14806.patch
>
>
> Introduced by HBASE-14085. The problem is, for example, in
> hbase-common/pom.xml, we have
> {code:title=pom.xml}
>
> org.apache.maven.plugins
> maven-source-plugin
>
> true
>
> src/main/java
> ${project.build.outputDirectory}/META-INF
>
>
>
> {code}
> But in fact, the path inside {{}} tag is relative to source
> directories, not the project directory. So the maven-source-plugin always end
> with
> {noformat}
> No sources in project. Archive not created.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15083) Gets from Multiactions are not counted in metrics for gets.
[ https://issues.apache.org/jira/browse/HBASE-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15083: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Gets from Multiactions are not counted in metrics for gets. > --- > > Key: HBASE-15083 > URL: https://issues.apache.org/jira/browse/HBASE-15083 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-15083-branch-1.patch, HBASE-15083.patch, > HBASE-15083.patch > > > RSRpcServices#get updates the get metrics. However Multiactions do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14869) Better request latency and size histograms
[ https://issues.apache.org/jira/browse/HBASE-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14869: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Better request latency and size histograms > -- > > Key: HBASE-14869 > URL: https://issues.apache.org/jira/browse/HBASE-14869 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Fix For: 2.0.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: 14869-test-0.98.txt, 14869-v1-0.98.txt, > 14869-v1-2.0.txt, 14869-v2-0.98.txt, 14869-v2-2.0.txt, 14869-v3-0.98.txt, > 14869-v4-0.98.txt, 14869-v5-0.98.txt, 14869-v6-0.98.txt, AppendSizeTime.png, > Get.png > > > I just discussed this with a colleague. > The get, put, etc, histograms that each region server keeps are somewhat > useless (depending on what you want to achieve of course), as they are > aggregated and calculated by each region server. > It would be better to record the number of requests in certainly latency > bands in addition to what we do now. > For example the number of gets that took 0-5ms, 6-10ms, 10-20ms, 20-50ms, > 50-100ms, 100-1000ms, > 1000ms, etc. (just as an example, should be > configurable). > That way we can do further calculations after the fact, and answer questions > like: How often did we miss our SLA? Percentage of requests that missed an > SLA, etc. > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14840) Sink cluster reports data replication request as success though the data is not replicated
[ https://issues.apache.org/jira/browse/HBASE-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14840: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Sink cluster reports data replication request as success though the data is > not replicated > -- > > Key: HBASE-14840 > URL: https://issues.apache.org/jira/browse/HBASE-14840 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.1.2, 1.0.3, 0.98.16 >Reporter: Y. SREENIVASULU REDDY >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: HBASE-14840-0.98.patch, HBASE-14840-branch-1.patch, > HBASE-14840.patch > > > *Scenario:* > Sink cluster is down > Create a table and enable table replication > Put some data > Now restart the sink cluster > *Observance:* > Data is not replicated in sink cluster but still source cluster updates the > WAL log position in ZK, resulting in data loss in sink cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15095) isReturnResult=false on fast path in branch-1.1 and branch-1.0 is not respected
[ https://issues.apache.org/jira/browse/HBASE-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15095: -- Fix Version/s: (was: 1.0.4) 1.0.3 > isReturnResult=false on fast path in branch-1.1 and branch-1.0 is not > respected > > > Key: HBASE-15095 > URL: https://issues.apache.org/jira/browse/HBASE-15095 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 1.1.2, 1.0.3 >Reporter: stack >Assignee: Heng Chen > Fix For: 1.0.3, 1.1.3 > > Attachments: HBASE-15095-branch-1.0.patch, > HBASE-15095-branch-1.0.patch, HBASE-15095-branch-1.1.patch > > > We don't pay attention to the isReturnResult when we go the fast path > increment. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14730) region server needs to log warnings when there are attributes configured for cells with hfile v2
[ https://issues.apache.org/jira/browse/HBASE-14730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14730: -- Fix Version/s: (was: 1.0.4) 1.0.3 > region server needs to log warnings when there are attributes configured for > cells with hfile v2 > > > Key: HBASE-14730 > URL: https://issues.apache.org/jira/browse/HBASE-14730 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.0.0, 1.1.0, 1.2.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 1.2.0, 1.0.3, 1.1.3 > > Attachments: HBASE-14730-v001-1.2.0.patch, HBASE-14730-v001.patch, > HBASE-14730-v002-1.2.0.patch, HBASE-14730-v002_branch-1.2.patch > > > User can configure cell attributes with hfile.format.version 2. When cells > are flushed from memStore to Hfile, attributes are not saved. Warnings need > to be logged. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15033) Backport test-patch.sh and zombie-detector.sh from master to branch-1.0/1.1
[ https://issues.apache.org/jira/browse/HBASE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15033: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Backport test-patch.sh and zombie-detector.sh from master to branch-1.0/1.1 > --- > > Key: HBASE-15033 > URL: https://issues.apache.org/jira/browse/HBASE-15033 > Project: HBase > Issue Type: Task > Components: build >Reporter: stack >Assignee: stack > Fix For: 1.0.3, 1.1.3 > > Attachments: 15033.patch > > > Backport current test-patch.sh and zombie dettector to branch-1.0+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15104) Occasional failures due to NotServingRegionException in IT tests
[ https://issues.apache.org/jira/browse/HBASE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15104: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Occasional failures due to NotServingRegionException in IT tests > > > Key: HBASE-15104 > URL: https://issues.apache.org/jira/browse/HBASE-15104 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.2.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Fix For: 2.0.0, 0.94.28, 1.2.0, 1.0.3, 1.1.3 > > Attachments: HBASE-15104-v001.patch > > > IntegrationTestAcidGuarantees fails when trying to cleanup with > NotServerRegionExceptions giving up (after 36 attempts) . > 5/11/09 09:19:24 INFO client.AsyncProcess: #33, waiting for some tasks to > finish. Expected max=0, tasksInProgress=9 > 15/11/09 09:19:33 INFO client.AsyncProcess: #45, table=TestAcidGuarantees, > attempt=10/35 failed=1ops, last exception: > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Region > TestAcidGuarantees,test_row_1,1447089367019.032439ef4f3353cb894d20337ba043bc. > is not online on node-4.internal,22101,1447089152259 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2786) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:922) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1893) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > at java.lang.Thread.run(Thread.java:745) > ... > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed > after attempts=36, exceptions: > Mon Nov 09 09:19:53 PST 2015, null, java.net.SocketTimeoutException: > callTimeout=6, callDuration=68104: row 'test_row_1' > Looked at the RS log, the following exception is found: > 2015-11-10 10:07:49,091 ERROR > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open > of > region=TestAcidGuarantees,,1447177733243.f1be6b850fe3958c5c9b5e330b5dfb00., > starting to roll back the global memstore size. > org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: com.hadoop.compression.lzo.LzoCodec > at > org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6011) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5995) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5967) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5938) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5894) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5845) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15108) TestReplicationAdmin failed on branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15108: -- Fix Version/s: (was: 1.0.4) 1.0.3 > TestReplicationAdmin failed on branch-1.0 > -- > > Key: HBASE-15108 > URL: https://issues.apache.org/jira/browse/HBASE-15108 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.3 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 1.0.3 > > Attachments: HBASE-15018-branch-1.0.patch, > HBASE-15108.v1.branch-1.0.patch > > > I notice it on HBASE-15095. > See > https://builds.apache.org/job/PreCommit-HBASE-Build/95/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14822) Renewing leases of scanners doesn't work
[ https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14822: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Renewing leases of scanners doesn't work > > > Key: HBASE-14822 > URL: https://issues.apache.org/jira/browse/HBASE-14822 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.14 >Reporter: Samarth Jain >Assignee: Lars Hofhansl > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.0.3, 1.1.3, 0.98.17 > > Attachments: 14822-0.98-addendum.txt, 14822-0.98-v2.txt, > 14822-0.98-v3.txt, 14822-0.98.txt, 14822-master-addendum.txt, > 14822-v3-0.98.txt, 14822-v4-0.98.txt, 14822-v4.txt, 14822-v5-0.98.txt, > 14822-v5-1.3.txt, 14822-v5.txt, 14822.txt, HBASE-14822_98_nextseq.diff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14923) VerifyReplication should not mask the exception during result comparison
[
https://issues.apache.org/jira/browse/HBASE-14923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14923:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> VerifyReplication should not mask the exception during result comparison
> -
>
> Key: HBASE-14923
> URL: https://issues.apache.org/jira/browse/HBASE-14923
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: 2.0.0, 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Minor
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-14923_v1.patch
>
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java
> Line:154
> } catch (Exception e) {
> logFailRowAndIncreaseCounter(context,
> Counters.CONTENT_DIFFERENT_ROWS, value);
> }
> Just LOG.error needs to be added for more information for the failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15052) Use EnvironmentEdgeManager in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15052:
--
Fix Version/s: (was: 1.0.4)
1.0.3
> Use EnvironmentEdgeManager in ReplicationSource
>
>
> Key: HBASE-15052
> URL: https://issues.apache.org/jira/browse/HBASE-15052
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-15052-v0.patch, HBASE-15052-v00.patch
>
>
> ReplicationSource is passing System.currentTimeMillis() to
> MetricsSource.setAgeOfLastShippedOp() which is subtracting that from
> EnvironmentEdgeManager.currentTime().
> {code}
> // if there was nothing to ship and it's not an error
> // set "ageOfLastShippedOp" to to indicate that we're current
> metrics.setAgeOfLastShippedOp(System.currentTimeMillis(), walGroupId);
> public void setAgeOfLastShippedOp(long timestamp, String walGroup) {
> long age = EnvironmentEdgeManager.currentTime() - timestamp;
> {code}
> we should just use EnvironmentEdgeManager.currentTime() in ReplicationSource
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14758) Add UT case for unchecked error/exception thrown in AsyncProcess#sendMultiAction
[ https://issues.apache.org/jira/browse/HBASE-14758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14758: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Add UT case for unchecked error/exception thrown in > AsyncProcess#sendMultiAction > > > Key: HBASE-14758 > URL: https://issues.apache.org/jira/browse/HBASE-14758 > Project: HBase > Issue Type: Test > Components: Client, test >Affects Versions: 1.1.2, 0.98.15 >Reporter: Yu Li >Assignee: Yu Li >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.4, 0.98.17 > > Attachments: HBASE-14758.branch-0.98.patch, HBASE-14758.patch > > > This is an addendum for HBASE-14359, open a new JIRA to trigger HadoopQA run -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15075) Allow region split request to carry identification information
[ https://issues.apache.org/jira/browse/HBASE-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15075: --- Status: Open (was: Patch Available) > Allow region split request to carry identification information > -- > > Key: HBASE-15075 > URL: https://issues.apache.org/jira/browse/HBASE-15075 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0, 1.3.0 > > Attachments: 15075-v0.txt, 15075-v1.txt, 15075-v2.txt, > HBASE-15075.v10.patch, HBASE-15075.v2.patch, HBASE-15075.v3.patch, > HBASE-15075.v4.patch, HBASE-15075.v5.patch, HBASE-15075.v6.patch, > HBASE-15075.v7.patch, HBASE-15075.v8.patch, HBASE-15075.v9.patch > > > During the process of improving region normalization feature, I found that if > region split request triggered by the execution of SplitNormalizationPlan > fails, there is no way of knowing whether the failed split originated from > region normalization. > The association of particular split request with outcome of split would give > RegionNormalizer information so that it can make better normalization > decisions in the subsequent invocations. > One approach is to embed metadata, such as a UUID, in SplitRequest which gets > passed through RegionStateTransitionContext when > RegionServerServices#reportRegionStateTransition() is called. > This way, RegionStateListener can be notified with the metadata (id of the > requester). > See discussion on dev mailing list > http://search-hadoop.com/m/YGbbCXdkivihp2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14845) hbase-server leaks jdk.tools dependency to mapreduce consumers
[ https://issues.apache.org/jira/browse/HBASE-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14845: -- Fix Version/s: (was: 1.0.3) 1.0.4 > hbase-server leaks jdk.tools dependency to mapreduce consumers > -- > > Key: HBASE-14845 > URL: https://issues.apache.org/jira/browse/HBASE-14845 > Project: HBase > Issue Type: Bug > Components: build, dependencies >Affects Versions: 2.0.0, 0.98.14, 1.2.0, 1.1.2, 1.3.0, 1.0.3 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18 > > > HBASE-13963 / HBASE-14844 take care of removing leaks of our dependency on > jdk-tools. > Until we move the mapreduce support classes out of hbase-server > (HBASE-11843), we need to also avoid leaking the dependency from that module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12557) Introduce timeout mechanism for IP to rack resolution
[ https://issues.apache.org/jira/browse/HBASE-12557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-12557. Resolution: Later Doesn't seem necessary at the moment > Introduce timeout mechanism for IP to rack resolution > - > > Key: HBASE-12557 > URL: https://issues.apache.org/jira/browse/HBASE-12557 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Labels: loadbalancer > Attachments: 12557-v1.txt, 12557-v3.txt > > > Config parameter, "hbase.util.ip.to.rack.determiner", determines the class > which does IP to rack resolution. > The actual resolution may be lengthy. > This JIRA is continuation of HBASE-12554 where a mock DNSToSwitchMapping is > used for rack resolution. > A timeout parameter, "hbase.ip.to.rack.determiner.timeout", is proposed whose > value governs the duration which RackManager waits before rack resolution is > stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15086) Fix findbugs complaint so yetus reports more green
[ https://issues.apache.org/jira/browse/HBASE-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15086: -- Attachment: none_fix.txt Try a none fix to see if findbugs are all cleaned up. > Fix findbugs complaint so yetus reports more green > -- > > Key: HBASE-15086 > URL: https://issues.apache.org/jira/browse/HBASE-15086 > Project: HBase > Issue Type: Umbrella > Components: build >Reporter: stack > Attachments: none_fix.txt > > > Umbrella issue for findbugs fixings. The red complaints in yetus report look > ugly. Lets fix. There ain't too many. This is umbrella issue. Can do a > component at a time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15118: -- Resolution: Fixed Assignee: stack Fix Version/s: 1.3.0 1.2.0 2.0.0 Status: Resolved (was: Patch Available) I pushed this to 1.2+ > Fix findbugs complaint in hbase-server > -- > > Key: HBASE-15118 > URL: https://issues.apache.org/jira/browse/HBASE-15118 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 15118.patch, 15118v2.patch, 15118v3.patch, 15118v4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15086) Fix findbugs complaint so yetus reports more green
[ https://issues.apache.org/jira/browse/HBASE-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15086: -- Assignee: stack Status: Patch Available (was: Open) > Fix findbugs complaint so yetus reports more green > -- > > Key: HBASE-15086 > URL: https://issues.apache.org/jira/browse/HBASE-15086 > Project: HBase > Issue Type: Umbrella > Components: build >Reporter: stack >Assignee: stack > Attachments: none_fix.txt > > > Umbrella issue for findbugs fixings. The red complaints in yetus report look > ugly. Lets fix. There ain't too many. This is umbrella issue. Can do a > component at a time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15107) Procedure V2 - Procedure Queue with Regions
[ https://issues.apache.org/jira/browse/HBASE-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106043#comment-15106043 ] Matteo Bertozzi commented on HBASE-15107: - forgot that this patch depends on HBASE-15106 (that's why it does not apply) > Procedure V2 - Procedure Queue with Regions > --- > > Key: HBASE-15107 > URL: https://issues.apache.org/jira/browse/HBASE-15107 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-15107-v0.patch > > > Adds the "region locking" that will be used to perform assign/unassign, > split/merge operations. > An operation take the xlock on the regions is working on, all the other > procedures will be suspended (removed from the runnable queue) and resumed > (put back in the runnable queue) when the operation that has the lock on the > region is completed. > https://reviews.apache.org/r/42213/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reopened HBASE-15073:
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15073:
---
Fix Version/s: (was: 1.3.0)
(was: 1.2.0)
(was: 2.0.0)
Release Note: (was: The NORMALIZATION_ENABLED_KEY attribute for table
has been renamed NORMALIZATION_MODE whose value represents the types of action
allowed for normalization.
To enable normalization for the table, you can specify 'M' for merging, 'S' for
splitting or "MS" for splitting / merging.
Leave empty to disable normalization.)
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[ https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106092#comment-15106092 ] chenrongwei commented on HBASE-15126: - OK,At first I added a unit test case,however according to the building tips like "Patch causes 11 errors with Hadoop v2.4.0." which I don't understand,so i removed the unit test case. > HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was > incorrect. > --- > > Key: HBASE-15126 > URL: https://issues.apache.org/jira/browse/HBASE-15126 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch > > > HBaseFsck's checkRegionBoundaries function set the > 'currentRegionBoundariesInformation.storesFirstKey' was incorrect.I think it > should be set like below, > currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey); > but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just > set to 'storeFirstKey',which will cause to > comparator.compare(currentRegionBoundariesInformation.storesFirstKey, > currentRegionBoundariesInformation.metaFirstKey) get a wrong > result. Because it just compared the rowkey's length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106140#comment-15106140 ] Hudson commented on HBASE-15118: SUCCESS: Integrated in HBase-1.3 #499 (See [https://builds.apache.org/job/HBase-1.3/499/]) HBASE-15118 Fix findbugs complaint in hbase-server (stack: rev d6654ea7ea6e8bb4f6d1b707c51f0fea0a289aa6) * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/SortedList.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitTransactionCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkRegionMergeCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/UserQuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/PrettyPrinter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ZKDataMigrator.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LruHashMap.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HashTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java
[jira] [Commented] (HBASE-15107) Procedure V2 - Procedure Queue with Regions
[
https://issues.apache.org/jira/browse/HBASE-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106038#comment-15106038
]
Hadoop QA commented on HBASE-15107:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-15107 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12782203/HBASE-15107-v0.patch |
| JIRA Issue | HBASE-15107 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/179/console |
This message was automatically generated.
> Procedure V2 - Procedure Queue with Regions
> ---
>
> Key: HBASE-15107
> URL: https://issues.apache.org/jira/browse/HBASE-15107
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-15107-v0.patch
>
>
> Adds the "region locking" that will be used to perform assign/unassign,
> split/merge operations.
> An operation take the xlock on the regions is working on, all the other
> procedures will be suspended (removed from the runnable queue) and resumed
> (put back in the runnable queue) when the operation that has the lock on the
> region is completed.
> https://reviews.apache.org/r/42213/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105962#comment-15105962
]
Enis Soztutar commented on HBASE-15073:
---
Maybe we can revert back to the normalization enabled, but also include two
parameters like "hbase.normalization.split.enabled" and
"hbase.normalization.merge.enabled" as advanced options. These still can be
table level options.
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15082: -- Attachment: 15082v14.patch Fix a few things. > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: 15082.patch, 15082v10.patch, 15082v12.patch, > 15082v13.patch, 15082v14.patch, 15082v14.patch, 15082v2.patch, 15082v2.txt, > 15082v3.txt, 15082v4.patch, 15082v5.patch, 15082v6.patch, 15082v7.patch, > 15082v8.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106015#comment-15106015
]
stack commented on HBASE-15073:
---
-1 on this change. Revert it. It is rubbish. When a proper case is made, I'll
change my vote. I can't make sense of the above (We split because region is
big... later, because of TTL, region is empty so we merge ... but this
'increases the work'... and "I talked with an Ambari dev". and again
'various use cases' but none but the mangled one specified)
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106179#comment-15106179
]
Hudson commented on HBASE-15073:
SUCCESS: Integrated in HBase-1.3-IT #442 (See
[https://builds.apache.org/job/HBase-1.3-IT/442/])
HBASE-15073 Revert due to different opinion on usefulness (tedyu: rev
7a096ac6cb402f53a092362fb1aa25d2a31f38fa)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* hbase-shell/src/main/ruby/shell/commands/normalize.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
* hbase-shell/src/main/ruby/shell/commands/normalizer_switch.rb
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/normalizer/NormalizationPlan.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen reassigned HBASE-15128: - Assignee: Heng Chen > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15128) Disable region splits and merges in HBCK
Enis Soztutar created HBASE-15128: - Summary: Disable region splits and merges in HBCK Key: HBASE-15128 URL: https://issues.apache.org/jira/browse/HBASE-15128 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Fix For: 2.0.0, 1.3.0 In large clusters where region splits are frequent, and HBCK runs take longer, the concurrent splits cause further problems in HBCK since HBCK assumes a static state for the region partition map. We have just seen a case where HBCK undo's a concurrently splitting region causing number of inconsistencies to go up. We can have a mode in master where splits and merges are disabled like the balancer and catalog janitor switches. Master will reject the split requests if regionservers decide to split. This switch can be turned on / off by the admins and also automatically by HBCK while it is running (similar to balancer switch being disabled by HBCK). HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106173#comment-15106173 ] Heng Chen commented on HBASE-15128: --- Sounds good! If you have not prepared for it, i can take it. [~enis] :) > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106174#comment-15106174 ] Enis Soztutar commented on HBASE-15128: --- Cool. Go for it! > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106184#comment-15106184 ] Hudson commented on HBASE-15118: FAILURE: Integrated in HBase-Trunk_matrix #641 (See [https://builds.apache.org/job/HBase-Trunk_matrix/641/]) HBASE-15118 Fix findbugs complaint in hbase-server (stack: rev 9bf26f46d1959e732a68d4e057c9a99701e5a047) * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/ExpiredMobFileCleanerChore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/SizeCachedKeyValue.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/SortedList.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HashTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/UserQuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionMover.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LruHashMap.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mob/ExpiredMobFileCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java > Fix findbugs complaint in hbase-server > -- > >
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106202#comment-15106202 ] Hudson commented on HBASE-15118: FAILURE: Integrated in HBase-1.2-IT #400 (See [https://builds.apache.org/job/HBase-1.2-IT/400/]) HBASE-15118 Fix findbugs complaint in hbase-server (stack: rev c4bcaa3f65da98c7f513c028d6e3639c8016c2f8) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HashTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/SortedList.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitTransactionCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkRegionMergeCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/UserQuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/PrettyPrinter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LruHashMap.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ZKDataMigrator.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java *
[jira] [Commented] (HBASE-15086) Fix findbugs complaint so yetus reports more green
[
https://issues.apache.org/jira/browse/HBASE-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106221#comment-15106221
]
Hadoop QA commented on HBASE-15086:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/latest/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 5s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 21s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 40s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 0s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 22s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 42s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 4s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 24s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 46s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 7s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 30s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 101m 10s
{color} | {color:green} root in the patch passed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 98m 41s
{color} | {color:green} root in the patch
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106236#comment-15106236
]
stack commented on HBASE-15073:
---
bq. ...different opinion on usefulness...
No.
The -1 is because no case was made for why we need this feature. Usually folks
show up w/ some numbers or a graph or some reasoning as to why something is
needed. In here there is none of that; there is 'certain use cases' that go
unspecified and then when pushed, some rambling about ambari metrics server
with no timelines or measure of what is being addressed or fit criteria as to
what would make ambari-case happy.
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-14221:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.0.4
1.1.3
1.3.0
1.2.0
Status: Resolved (was: Patch Available)
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 1.0.4
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221-branch-1.patch,
> HBASE-14221.patch, HBASE-14221_1.patch, HBASE-14221_1.patch,
> HBASE-14221_6.patch, HBASE-14221_9.patch, withmatchingRowspatch.png,
> withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106278#comment-15106278
]
Hudson commented on HBASE-15073:
FAILURE: Integrated in HBase-1.2 #511 (See
[https://builds.apache.org/job/HBase-1.2/511/])
HBASE-15073 Revert due to different opinion on usefulness (tedyu: rev
5ff65455c60dbf56fc92630e75d93d7840dde537)
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
* hbase-shell/src/main/ruby/shell/commands/normalizer_switch.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/normalizer/NormalizationPlan.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
* hbase-shell/src/main/ruby/hbase/admin.rb
* hbase-shell/src/main/ruby/shell/commands/normalize.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106279#comment-15106279 ] Hudson commented on HBASE-15118: FAILURE: Integrated in HBase-1.2 #511 (See [https://builds.apache.org/job/HBase-1.2/511/]) HBASE-15118 Fix findbugs complaint in hbase-server (stack: rev c4bcaa3f65da98c7f513c028d6e3639c8016c2f8) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/SortedList.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LruHashMap.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkRegionMergeCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitTransactionCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/UserQuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ZKDataMigrator.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/PrettyPrinter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java *
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106284#comment-15106284
]
Anoop Sam John commented on HBASE-15097:
Actually whatever be the stopRow value being set, the under layers of scan
should not give a row outside the regions boundary. IMHO, we should investigate
how that got broken and fix that issue. If the scan is not specifying any
stopRow and because of the said bug, an out of boundary row can come out right?
We should fix the root cause.
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106149#comment-15106149
]
Heng Chen commented on HBASE-15122:
---
relates testcase has passed.
https://builds.apache.org/job/PreCommit-HBASE-Build/167/testReport/org.apache.hadoop.hbase.http.jmx/TestJMXJsonServlet/
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was incorrect.
[ https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15126: Attachment: HBASE-15126-v003.patch Add an unit test case for region boundary check. > HBaseFsck's checkRegionBoundaries function set the 'storesFirstKey' was > incorrect. > --- > > Key: HBASE-15126 > URL: https://issues.apache.org/jira/browse/HBASE-15126 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, > HBASE-15126-v003.patch > > > HBaseFsck's checkRegionBoundaries function set the > 'currentRegionBoundariesInformation.storesFirstKey' was incorrect.I think it > should be set like below, > currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey); > but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just > set to 'storeFirstKey',which will cause to > comparator.compare(currentRegionBoundariesInformation.storesFirstKey, > currentRegionBoundariesInformation.metaFirstKey) get a wrong > result. Because it just compared the rowkey's length. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK
[ https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106192#comment-15106192 ] Vladimir Rodionov commented on HBASE-15128: --- It is not only for hbck. Snapshots and bulk load will benefit from disabling splits as well. > Disable region splits and merges in HBCK > > > Key: HBASE-15128 > URL: https://issues.apache.org/jira/browse/HBASE-15128 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0 > > > In large clusters where region splits are frequent, and HBCK runs take > longer, the concurrent splits cause further problems in HBCK since HBCK > assumes a static state for the region partition map. We have just seen a case > where HBCK undo's a concurrently splitting region causing number of > inconsistencies to go up. > We can have a mode in master where splits and merges are disabled like the > balancer and catalog janitor switches. Master will reject the split requests > if regionservers decide to split. This switch can be turned on / off by the > admins and also automatically by HBCK while it is running (similar to > balancer switch being disabled by HBCK). > HBCK should also disable the Catalog Janitor just in case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106201#comment-15106201
]
Hudson commented on HBASE-15073:
FAILURE: Integrated in HBase-1.2-IT #400 (See
[https://builds.apache.org/job/HBase-1.2-IT/400/])
HBASE-15073 Revert due to different opinion on usefulness (tedyu: rev
5ff65455c60dbf56fc92630e75d93d7840dde537)
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/normalizer/NormalizationPlan.java
* hbase-shell/src/main/ruby/shell/commands/normalize.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
* hbase-shell/src/main/ruby/hbase/admin.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
* hbase-shell/src/main/ruby/shell/commands/normalizer_switch.rb
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15086) Fix findbugs complaint so yetus reports more green
[ https://issues.apache.org/jira/browse/HBASE-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15086: -- Attachment: none_fix.branch-1.patch Try branch-1 > Fix findbugs complaint so yetus reports more green > -- > > Key: HBASE-15086 > URL: https://issues.apache.org/jira/browse/HBASE-15086 > Project: HBase > Issue Type: Umbrella > Components: build >Reporter: stack >Assignee: stack > Attachments: none_fix.branch-1.patch, none_fix.txt > > > Umbrella issue for findbugs fixings. The red complaints in yetus report look > ugly. Lets fix. There ain't too many. This is umbrella issue. Can do a > component at a time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15085) IllegalStateException was thrown when scanning on bulkloaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106270#comment-15106270
]
Anoop Sam John commented on HBASE-15085:
HBASE-15104 is fixed now.. So this issue patch is no way related to that
failure. Just commented for the conclusion.
> IllegalStateException was thrown when scanning on bulkloaded HFiles
> ---
>
> Key: HBASE-15085
> URL: https://issues.apache.org/jira/browse/HBASE-15085
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.12, 1.1.2
> Environment: HBase-0.98.12 & Hadoop-2.6.0 & JDK1.7
> HBase-1.1.2 & Hadoop-2.6.0 & JDK1.7
>Reporter: Victor Xu
>Assignee: Victor Xu
>Priority: Critical
> Labels: hfile
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17
>
> Attachments: HBASE-15085-0.98-v1.patch, HBASE-15085-0.98-v2.patch,
> HBASE-15085-0.98-v3.patch, HBASE-15085-0.98-v4.patch,
> HBASE-15085-0.98-v4.patch, HBASE-15085-0.98-v5.patch,
> HBASE-15085-branch-1.0-v1.patch, HBASE-15085-branch-1.0-v2.patch,
> HBASE-15085-branch-1.1-v1.patch, HBASE-15085-branch-1.1-v2.patch,
> HBASE-15085-branch-1.2-v1.patch, HBASE-15085-branch-1.2-v2.patch,
> HBASE-15085-v1.patch, HBASE-15085-v2.patch, HBASE-15085-v3.patch,
> HBASE-15085-v4.patch, HBASE-15085-v4.patch, HBASE-15085-v4.patch,
> HBASE-15085-v4.patch, HBASE-15085-v5.patch
>
>
> IllegalStateException was thrown when we scanned from an HFile which was bulk
> loaded several minutes ago, as shown below:
> {code}
> 2015-12-16 22:20:54,456 ERROR
> com.taobao.kart.coprocessor.server.KartCoprocessor:
> icbu_ae_ws_product,/0055,1450275490479.6a6a700f465ad074287fed720c950f7c.
> batchNotify exception
> java.lang.IllegalStateException: EncodedScanner works only on encoded data
> blocks
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.updateCurrentBlock(HFileReaderV2.java:1042)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.seekTo(HFileReaderV2.java:1093)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:244)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:329)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:188)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1879)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:4068)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2029)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2015)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1992)
> {code}
> I used 'hbase hfile' command to analyse the meta and block info of the hfile,
> finding that even through the DATA_BLOCK_ENCODING was 'DIFF' in FileInfo, the
> actual data blocks was written without any encoding algorithms(BlockType was
> 'DATA', not 'ENCODED_DATA'):
> {code}
> Fileinfo:
> BLOOM_FILTER_TYPE = ROW
> BULKLOAD_SOURCE_TASK = attempt_1442077249005_606706_r_12_0
> BULKLOAD_TIMESTAMP = \x00\x00\x01R\x12$\x13\x12
> DATA_BLOCK_ENCODING = DIFF
> ...
> DataBlock Header:
> HFileBlock [ fileOffset=0 headerSize()=33 blockType=DATA
> onDiskSizeWithoutHeader=65591 uncompressedSizeWithoutHeader=65571
> prevBlockOffset=-1 isUseHBaseChecksum()=true checksumType=CRC32
> bytesPerChecksum=16384 onDiskDataSizeWithHeader=65604
> getOnDiskSizeWithHeader()=65624 totalChecksumBytes()=20 isUnpacked()=true
> buf=[ java.nio.HeapByteBuffer[pos=0 lim=65624 cap=65657],
> array().length=65657, arrayOffset()=0 ]
> dataBeginsWith=\x00\x00\x003\x00\x00\x00\x0A\x00\x10/0008:18\x01dprod
> fileContext=HFileContext [ usesHBaseChecksum=true checksumType=CRC32
> bytesPerChecksum=16384 blocksize=65536 encoding=NONE includesMvcc=true
> includesTags=false compressAlgo=NONE compressTags=false cryptoContext=[
> cipher=NONE keyHash=NONE ] ] ]
> {code}
> The data block encoding in file info was not consistent with the one in data
> block, which means there must be something wrong with the bulkload process.
> After debugging on each step of bulkload, I found that LoadIncrementalHFiles
> had a bug when loading hfile into a splitted region.
> {code}
> /**
>* Copy half of an HFile into a new HFile.
>*/
> private static void copyHFileHalf(
> Configuration conf, Path inFile, Path outFile, Reference reference,
> HColumnDescriptor familyDescriptor)
> throws IOException {
> FileSystem fs =
[jira] [Created] (HBASE-15127) NPE in RpcServer$Call.wrapWithSasl()
Enis Soztutar created HBASE-15127:
-
Summary: NPE in RpcServer$Call.wrapWithSasl()
Key: HBASE-15127
URL: https://issues.apache.org/jira/browse/HBASE-15127
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Fix For: 2.0.0, 1.3.0
I saw this in a log file, not sure whether it is important to fix or not:
{code}
2016-01-09 01:30:58,905 WARN [FifoRpcScheduler.handler1-thread-25]
ipc.RpcServer: FifoRpcScheduler.handler1-thread-25: caught:
java.lang.NullPointerException
at org.apache.hadoop.hbase.ipc.RpcServer$Call.wrapWithSasl(RpcServer.java:412)
at org.apache.hadoop.hbase.ipc.RpcServer$Call.setResponse(RpcServer.java:395)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:128)
at
org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
This is 0.98 based code base corresponding to:
{code}
synchronized (connection.saslServer) {
token = connection.saslServer.wrap(responseBytes, 0,
responseBytes.length);
}
{code}
I believe the saslserver was set to null earlier by {{disposeSasl()}} because
Connection is closing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15118) Fix findbugs complaint in hbase-server
[ https://issues.apache.org/jira/browse/HBASE-15118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106180#comment-15106180 ] Hudson commented on HBASE-15118: SUCCESS: Integrated in HBase-1.3-IT #442 (See [https://builds.apache.org/job/HBase-1.3-IT/442/]) HBASE-15118 Fix findbugs complaint in hbase-server (stack: rev d6654ea7ea6e8bb4f6d1b707c51f0fea0a289aa6) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ZKDataMigrator.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LruHashMap.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkRegionMergeCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/SortedList.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HashTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreChunkPool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/PrettyPrinter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/UserQuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaState.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/IdReadWriteLock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityController.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitTransactionCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableNamespaceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java *
[jira] [Updated] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function create region with wrong end key boundary.
[ https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenrongwei updated HBASE-15125: Attachment: HBASE-15125-v002.patch Add an unit test case for this patch > HBaseFsck's adoptHdfsOrphan function create region with wrong end key > boundary. > --- > > Key: HBASE-15125 > URL: https://issues.apache.org/jira/browse/HBASE-15125 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 2.0.0 >Reporter: chenrongwei >Assignee: chenrongwei > Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch > > > There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this > function will create a region,which want to cover all the orphan regions.But > the end key of this new region was set incorrectly.Correct region's boundary > should be [startKey,endKey),but this function create a region with boundary > of [startKey,endKey],this bug will leads to scan operation omit some data. > I think we should create the region like bellow, > // create new region on hdfs. move data into place. > HRegionInfo hri = new HRegionInfo(template.getTableName(), > orphanRegionRange.getFirst(), > Bytes.add(orphanRegionRange.getSecond(), new byte[1])); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[ https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106277#comment-15106277 ] Anoop Sam John commented on HBASE-9393: --- We open all HFiles in an RS and will be doing a read of the FileInfo. I believe this is not a pread. I believe we can change it to do pread and try. #2 is also ok after we read the FileInfo. bq.The above implementation will be configurable and by default disabled as we are expecting some impact on read flow. Closing the HFile's inputStreams is not a good option because of its impact. > Hbase does not closing a closed socket resulting in many CLOSE_WAIT > > > Key: HBASE-9393 > URL: https://issues.apache.org/jira/browse/HBASE-9393 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2, 0.98.0 > Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, > 7279 regions >Reporter: Avi Zrachya > > HBase dose not close a dead connection with the datanode. > This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect > to the datanode because too many mapped sockets from one host to another on > the same port. > The example below is with low CLOSE_WAIT count because we had to restart > hbase to solve the porblem, later in time it will incease to 60-100K sockets > on CLOSE_WAIT > [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l > 13156 > [root@hd2-region3 ~]# ps -ef |grep 21592 > root 17255 17219 0 12:26 pts/000:00:00 grep 21592 > hbase21592 1 17 Aug29 ?03:29:06 > /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m > -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -Dhbase.log.dir=/var/log/hbase > -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106309#comment-15106309
]
chenrongwei commented on HBASE-15097:
-
Yes,you're right. we should find and fix the root cause finally, and it's more
important to us actually. but I think it's still important to avoid the bug by
setting the correct stop row and it's more easier done. For example, if we
don't do this,maybe it's more difficult to find bugs like HBASE-15125.
Additionally,"If the scan is not specifying any stopRow and because of the said
bug, an out of boundary row can come out right? ",I said no,if we set the
correct stop row,then it will not happen,because the error will not happen at
the last region.
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.
[
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106315#comment-15106315
]
chenrongwei commented on HBASE-15097:
-
I think HBASE-15125 will cause data out of its boundary,but obviously it's not
the only way.
> When the scan operation covered two regions,sometimes the final results have
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch,
> output.log, rowkey.txt, snapshot2016-01-13 pm 8.42.37.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first
> region returned the rows which were beyond of its' end key.So,this finally
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) &&
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(),
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15101) Leaked References to StoreFile.Reader after HBASE-13082
[ https://issues.apache.org/jira/browse/HBASE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106396#comment-15106396 ] ramkrishna.s.vasudevan commented on HBASE-15101: Patch looks good to me. ya I think the shorter version is the easiest way after seeing the code where the patch is prepared. Regarding the point where you say a Region Server dies and you end up seeing lot of uncollected files, in the version without HBASE-13082 - suppose a set of compactions started but that RS got killed or dies before it is completed you wil have all those store files again getting used for subsequent readers when the regions are opened in a new RS and there should be another compaction that needs to run and move the files to the archive dir. Now even after HBASE-13082 - the scenario is similar because once the compaction is done if the Compaction discharger thread does not kick in and before that your RS dies, you end up with the store files again available for reads and you need one more set of compaction to happen in the new RS to move the files to the archive dir. After applying the patch also you still get issues where the compacted files are not cleared? You mean there is always a set of files that is not cleared ? Now coming to do the cleaning operation during close(), I think we are doing it on HStore.close() operation and ensuring that the compacted files are moved to the archive dir. Its good to check this out before we commit the updated patch to branch-1. > Leaked References to StoreFile.Reader after HBASE-13082 > --- > > Key: HBASE-15101 > URL: https://issues.apache.org/jira/browse/HBASE-15101 > Project: HBase > Issue Type: Bug > Components: HFile, io >Affects Versions: 2.0.0 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-15101-v1.patch, HBASE-15101-v2.patch, > HBASE-15101-v3.patch, HBASE-15101.patch > > > We observed this production that after a region server dies there are huge > number of hfiles in that region for the region server running the version > with HBASE-13082, In the doc it is given that it is expected to happen, but > we found a one place where scanners are not being closed. If the scanners are > not closed their references are not decremented and that is leading to the > issue of huge number of store files not being finalized > All I was able to find is in the selectScannersFrom, where we discard some of > the scanners and we are not closing them. I am attaching a patch for that. > Also to avoid these issues should the files that are done be logged and > finalized (moved to archive) as a part of region close operation. This will > solve any leaks that can happen and does not cause any dire consequences? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106296#comment-15106296
]
Hudson commented on HBASE-15073:
SUCCESS: Integrated in HBase-1.3 #500 (See
[https://builds.apache.org/job/HBase-1.3/500/])
HBASE-15073 Revert due to different opinion on usefulness (tedyu: rev
7a096ac6cb402f53a092362fb1aa25d2a31f38fa)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
* hbase-shell/src/main/ruby/shell/commands/normalize.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
* hbase-shell/src/main/ruby/hbase/admin.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/normalizer/NormalizationPlan.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
* hbase-shell/src/main/ruby/shell/commands/normalizer_switch.rb
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15091) Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"
[ https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15091: -- Attachment: 15091v8.branch-1.2.patch This seems to fix the TestAtomicOperation... HBASE-12751 makes this stuff work different. Let me run some comparisons. Our 'fast path' may not be so 'fast path' with this forward ported patch. The checkstyle I can't/won't fix. It is complaint that the append method is 186 lines long when limit is 150; its fixed in the master version of the patch. > Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and SequenceID performance > regression in branch-1.0" > --- > > Key: HBASE-15091 > URL: https://issues.apache.org/jira/browse/HBASE-15091 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Priority: Blocker > Fix For: 1.2.0 > > Attachments: 15091v2.branch-1.2.patch, 15091v3.branch-1.2.patch, > 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 15091v6.branch-1.2.patch, > 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 15091v8.branch-1.2.patch, > HBASE-15091-branch-1.2.patch, HBASE-15091-branch-1.2_v1.patch, > HBASE-15091.v1.branch-1.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15073) Finer grained control over normalization actions for RegionNormalizer
[
https://issues.apache.org/jira/browse/HBASE-15073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106400#comment-15106400
]
Hudson commented on HBASE-15073:
FAILURE: Integrated in HBase-Trunk_matrix #642 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/642/])
HBASE-15073 Revert due to different opinion on usefulness (tedyu: rev
eb17f74b9e45146a21584f44e7a811fedeee138f)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* hbase-shell/src/main/ruby/shell/commands/normalize.rb
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizerOnCluster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/normalizer/TestSimpleRegionNormalizer.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizer.java
* hbase-shell/src/main/ruby/hbase/admin.rb
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/NormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/MergeNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/EmptyNormalizationPlan.java
* hbase-shell/src/main/ruby/shell/commands/normalizer_switch.rb
* hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/normalizer/NormalizationPlan.java
> Finer grained control over normalization actions for RegionNormalizer
> -
>
> Key: HBASE-15073
> URL: https://issues.apache.org/jira/browse/HBASE-15073
> Project: HBase
> Issue Type: Task
> Components: regionserver
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 15073-v1.txt, 15073-v2.txt, 15073-v2.txt, 15073-v3.txt,
> 15073-v4.txt, 15073-v5.txt
>
>
> Currently both region split and merge actions are carried out during
> normalization for underlying table.
> However, for certain use case(s), it would be desirable to perform only one
> type of action.
> There is one boolean flag, keyed by NORMALIZATION_ENABLED_KEY, per table that
> enables normalization.
> To provide finer grained control, we have several options:
> 1. introduce another per table flag to indicate which type(s) of actions are
> allowed ("N" for disabled, "S" for split only, "M" for merge only and "MS"
> for both split and merge)
> 2. introduce another global flag to indicate which type(s) of actions are
> allowed
> 3. modify the meaning of existing flag keyed by NORMALIZATION_ENABLED_KEY so
> that it indicates type(s) of actions
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15056) Split fails with KeeperException$NoNodeException when namespace quota is enabled
[
https://issues.apache.org/jira/browse/HBASE-15056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-15056:
---
Status: Open (was: Patch Available)
> Split fails with KeeperException$NoNodeException when namespace quota is
> enabled
>
>
> Key: HBASE-15056
> URL: https://issues.apache.org/jira/browse/HBASE-15056
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Ted Yu
> Attachments: 15056-branch-1-v1.txt,
> split-fails-when-exceeding-quota-with-znode-loss.test
>
>
> When trying to port HBASE-15044 to branch-1, I found that region split fails
> with KeeperException$NoNodeException when namespace quota is enabled and the
> split would exceed allocated quota.
> Here is related test output:
> {code}
> 2015-12-30 09:50:16,764 WARN [RS:0;10.22.24.71:65256-splits-1451497816754]
> zookeeper.ZKAssign(885): regionserver:65256-0x151f402c21c0001,
> quorum=localhost:57662, baseZNode=/hbase Attempt to transition the
> unassigned node for 17fc99c04a8027b653e9d5ef5d578461 from
> RS_ZK_REQUEST_REGION_SPLIT to RS_ZK_REQUEST_REGION_SPLIT failed, the node
> existed and was in the expected state but then when setting data it no
> longer existed
> 2015-12-30 09:50:16,866 DEBUG [RS:0;10.22.24.71:65256-splits-1451497816754]
> zookeeper.ZKUtil(718): regionserver:65256-0x151f402c21c0001,
> quorum=localhost:57662, baseZNode=/hbase Unable to get data of znode
> /hbase/region-in-transition/17fc99c04a8027b653e9d5ef5d578461 because node
> does not exist (not necessarily an error)
> 2015-12-30 09:50:16,866 INFO [RS:0;10.22.24.71:65256-splits-1451497816754]
> regionserver.SplitRequest(97): Running rollback/cleanup of failed split of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451497806295.17fc99c04a8027b653e9d5ef5d578461.;
> Failed getting SPLITTING znode on
> np2:testRegionNormalizationSplitOnCluster,z,
> 1451497806295.17fc99c04a8027b653e9d5ef5d578461.
> java.io.IOException: Failed getting SPLITTING znode on
> np2:testRegionNormalizationSplitOnCluster,z,1451497806295.17fc99c04a8027b653e9d5ef5d578461.
> at
> org.apache.hadoop.hbase.coordination.ZKSplitTransactionCoordination.waitForSplitTransaction(ZKSplitTransactionCoordination.java:200)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:381)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:277)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:560)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:154)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Data is null, splitting node
> 17fc99c04a8027b653e9d5ef5d578461 no longer exists
> at
> org.apache.hadoop.hbase.coordination.ZKSplitTransactionCoordination.waitForSplitTransaction(ZKSplitTransactionCoordination.java:166)
> ... 8 more
> 2015-12-30 09:50:16,869 DEBUG [RS:0;10.22.24.71:65256-splits-1451497816754]
> zookeeper.ZKUtil(718): regionserver:65256-0x151f402c21c0001,
> quorum=localhost:57662, baseZNode=/hbase Unable to get data of znode
> /hbase/region-in-transition/17fc99c04a8027b653e9d5ef5d578461 because node
> does not exist (not necessarily an error)
> 2015-12-30 09:50:16,869 INFO [RS:0;10.22.24.71:65256-splits-1451497816754]
> coordination.ZKSplitTransactionCoordination(268): Failed cleanup zk node of
> np2:
> testRegionNormalizationSplitOnCluster,z,1451497806295.17fc99c04a8027b653e9d5ef5d578461.
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:452)
> at org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:381)
> at
> org.apache.hadoop.hbase.coordination.ZKSplitTransactionCoordination.clean(ZKSplitTransactionCoordination.java:261)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.rollback(SplitTransactionImpl.java:948)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.rollback(SplitTransactionImpl.java:900)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:99)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:154)
> at
>
[jira] [Updated] (HBASE-15031) Fix merge of MVCC and SequenceID performance regression in branch-1.0
[ https://issues.apache.org/jira/browse/HBASE-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15031: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Fix merge of MVCC and SequenceID performance regression in branch-1.0 > - > > Key: HBASE-15031 > URL: https://issues.apache.org/jira/browse/HBASE-15031 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 1.0.3 >Reporter: stack >Assignee: stack > Fix For: 1.0.3, 1.1.3 > > Attachments: 14460.v0.branch-1.0.patch, 15031.v2.branch-1.0.patch, > 15031.v3.branch-1.0.patch, 15031.v4.branch-1.0.patch, > 15031.v5.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v6.branch-1.0.patch, 15031.v6.branch-1.0.patch, > 15031.v7.branch-1.0.patch, 15031.v8.branch-1.0.patch, 15031.v8.master.patch > > > Subtask with fix for branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT
[
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105939#comment-15105939
]
Colin Patrick McCabe commented on HBASE-9393:
-
Unfortunately, this is kind of a complex topic.
In HDFS, sockets for input streams are managed by the {{Peer}} class.
{{Peers}} can either be "owned" by {{DFSInputStream}} objects, or stored in the
{{PeerCache}}. The {{PeerCache}} already has appropriate timeouts and won't
keep open too many sockets. However, there is no limit to how long a
{{DFSInputStream}} could hold on to a {{Peer}}.
There are a few ways to minimize the number of open peers.
1. If HBase only ever called positional read (pread), the {{DFSInputStream}}
object would never own a {{Peer}}, so this issue would not arise.
2. If HBase called {{DFSInputStream#unbuffer}}, any open peers would be closed,
even though the stream would continue to be open.
3. If HDFS had a timeout for how long it would hold onto a {{Peer}}, that could
limit the number of open sockets.
Configuring HBase to periodically close open streams is not necessary; it's
strictly worse than option #2.
I believe there is an option do to #1 even right now. Can't HBase be
configured just to use pread and never read? #2 would require a code change to
HBase; #3 would require a code change to HDFS.
Are you running out of file descriptors? What's the user-visible problem here?
> Hbase does not closing a closed socket resulting in many CLOSE_WAIT
>
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node,
> 7279 regions
>Reporter: Avi Zrachya
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect
> to the datanode because too many mapped sockets from one host to another on
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart
> hbase to solve the porblem, later in time it will incease to 60-100K sockets
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219 0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
> -Dhbase.log.dir=/var/log/hbase
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14940) Make our unsafe based ops more safe
[ https://issues.apache.org/jira/browse/HBASE-14940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14940: -- Fix Version/s: (was: 1.0.4) 1.0.3 > Make our unsafe based ops more safe > --- > > Key: HBASE-14940 > URL: https://issues.apache.org/jira/browse/HBASE-14940 > Project: HBase > Issue Type: Bug >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.17 > > Attachments: HBASE-14940.patch, HBASE-14940_addendum_0.98.patch, > HBASE-14940_branch-1.patch, HBASE-14940_branch-1.patch, > HBASE-14940_branch-1.patch, HBASE-14940_branch-1.patch, HBASE-14940_v2.patch > > > Thanks for the nice findings [~ikeda] > This jira solves 3 issues with Unsafe operations and ByteBufferUtils > 1. We can do sun unsafe based reads and writes iff unsafe package is > available and underlying platform is having unaligned-access capability. But > we were missing the second check > 2. Java NIO is doing a chunk based copy while doing Unsafe copyMemory. The > max chunk size is 1 MB. This is done for "A limit is imposed to allow for > safepoint polling during a large copy" as mentioned in comments in Bits.java. > We are also going to do same way > 3. In ByteBufferUtils, when Unsafe is not available and ByteBuffers are off > heap, we were doing byte by byte operation (read/copy). We can avoid this and > do better way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15092) HBase personality should order modules correctly
[
https://issues.apache.org/jira/browse/HBASE-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105828#comment-15105828
]
Ted Yu commented on HBASE-15092:
YETUS-280 has alleviated the dependency ordering issue.
> HBase personality should order modules correctly
>
>
> Key: HBASE-15092
> URL: https://issues.apache.org/jira/browse/HBASE-15092
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Ted Yu
>
> See the thread on dev@yetus, 'determining cause for compilation error'.
> In HBASE-15075, I modify class SplitNormalizationPlan.java in hbase-server
> module which calls the following new method added to Admin.java (in
> hbase-client module):
> {code}
> + void splitRegion(final byte[] regionName, final byte[] splitPoint, final
> UUID id)
> {code}
> Incorrect order of building the modules led to the following compilation
> error:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /testptch/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SplitNormalizationPlan.java:[95,12]
> no suitable method found for splitRegion(byte[],byte[],java.util.UUID)
> method org.apache.hadoop.hbase.client.Admin.splitRegion(byte[]) is not
> applicable
> (actual and formal argument lists differ in length)
> method org.apache.hadoop.hbase.client.Admin.splitRegion(byte[],byte[]) is
> not applicable
> (actual and formal argument lists differ in length)
> {code}
> HBase personality should order the modules according to the Maven
> dependencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15107) Procedure V2 - Procedure Queue with Regions
[ https://issues.apache.org/jira/browse/HBASE-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105911#comment-15105911 ] Ted Yu commented on HBASE-15107: If you name the patch HBASE-15107.v0.patch, it should be picked up by QA. > Procedure V2 - Procedure Queue with Regions > --- > > Key: HBASE-15107 > URL: https://issues.apache.org/jira/browse/HBASE-15107 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-15107-v0.patch > > > Adds the "region locking" that will be used to perform assign/unassign, > split/merge operations. > An operation take the xlock on the regions is working on, all the other > procedures will be suspended (removed from the runnable queue) and resumed > (put back in the runnable queue) when the operation that has the lock on the > region is completed. > https://reviews.apache.org/r/42213/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
