[jira] [Updated] (HBASE-9766) HFileV3 - Optional tags write and read is not working as expected
[ https://issues.apache.org/jira/browse/HBASE-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9766: -- Attachment: HBASE-9766_V3.patch Corrected the test failure. Let QA run on this. Once all tests pass I will commit. > HFileV3 - Optional tags write and read is not working as expected > - > > Key: HBASE-9766 > URL: https://issues.apache.org/jira/browse/HBASE-9766 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.98.0 > > Attachments: HBASE-9766.patch, HBASE-9766_V2.patch, > HBASE-9766_V3.patch > > > In the writer V3 includesTags always comes as true only and so writing tags > length always even when compaction selection says not to do so. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796461#comment-13796461 ] Vladimir Rodionov commented on HBASE-9769: -- Some additional performance numbers on a patch: Table: 1 CF + 5 CQ, value ~ 10-15 bytes. Rows = 50. All data in a block cache. Tested on *StoreScanner * directly. Default: Raw = 1.28M rows per sec 1 CQ in Scan = 0.7M 2 CQ in Scan = 0.5M 3 CQ in Scan = 0.4M 4 CQ in Scan = 0.32M 5 CQ in Scan = 0.33M Patch: Raw = 1.28M rows per sec 1 CQ in Scan = 1.27M 2 CQ in Scan = 1.2M 3 CQ in Scan = 1.1M 4 CQ in Scan = 1.05M 5 CQ in Scan = 1M > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[
https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796460#comment-13796460
]
Hadoop QA commented on HBASE-9778:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608659/9778-trunk.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestExplicitColumnTracker
org.apache.hadoop.hbase.regionserver.TestQueryMatcher
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7560//console
This message is automatically generated.
> Avoid seeking to next column in ExplicitColumnTracker when possible
> ---
>
> Key: HBASE-9778
> URL: https://issues.apache.org/jira/browse/HBASE-9778
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9778-0.94.txt, 9778-trunk.txt
>
>
> The issue of slow seeking in ExplicitColumnTracker was brought up by
> [~vrodionov] on the dev list.
> My idea here is to avoid the seeking if we know that there aren't many
> versions to skip.
> How do we know? We'll use the column family's VERSIONS setting as a hint. If
> VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call
> SKIP repeatedly.
> HBASE-9769 has some initial number for this approach:
> Interestingly it depends on which column(s) is (are) selected.
> Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1,
> everything filtered at the server with a ValueFilter. Everything measured in
> seconds.
> Without patch:
> ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4||
> |6.4|8.5|14.3|14.6|11.1|20.3|
> With patch:
> ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4||
> |6.4|8.4|8.9|9.9|6.4|10.0|
> Variation here was +- 0.2s.
> So with this patch scanning is 2x faster than without in some cases, and
> never slower. No special hint needed, beyond declaring VERSIONS correctly.
--
This message was sent by Atlassian JIRA
(v6.1#
[jira] [Commented] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[ https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796457#comment-13796457 ] stack commented on HBASE-9778: -- Nice numbers for such a little code change. I don't know this code well. Looking at it, doesn't SKIP mean 'do not include this KV in the result'? So if maxVersion == 1, it means we already have a KV and we are saying 'do not include this result' for all subsequent seen columns rather than call a seek? What if many versions of this row. Now we are not seeking, we could spend a bunch of time having to skip all values? Am I groking this right? > Avoid seeking to next column in ExplicitColumnTracker when possible > --- > > Key: HBASE-9778 > URL: https://issues.apache.org/jira/browse/HBASE-9778 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9778-0.94.txt, 9778-trunk.txt > > > The issue of slow seeking in ExplicitColumnTracker was brought up by > [~vrodionov] on the dev list. > My idea here is to avoid the seeking if we know that there aren't many > versions to skip. > How do we know? We'll use the column family's VERSIONS setting as a hint. If > VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call > SKIP repeatedly. > HBASE-9769 has some initial number for this approach: > Interestingly it depends on which column(s) is (are) selected. > Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, > everything filtered at the server with a ValueFilter. Everything measured in > seconds. > Without patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.5|14.3|14.6|11.1|20.3| > With patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.4|8.9|9.9|6.4|10.0| > Variation here was +- 0.2s. > So with this patch scanning is 2x faster than without in some cases, and > never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796454#comment-13796454
]
Devaraj Das commented on HBASE-9773:
Cool. Now it seems this will also fix HBASE-9777. Would it?
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796455#comment-13796455
]
Hudson commented on HBASE-9773:
---
SUCCESS: Integrated in hbase-0.96 #143 (See
[https://builds.apache.org/job/hbase-0.96/143/])
HBASE-9773 Master aborted when hbck asked the master to assign a region that
was already online (jxiang: rev 1532634)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796448#comment-13796448
]
Hudson commented on HBASE-9773:
---
SUCCESS: Integrated in HBase-TRUNK #4622 (See
[https://builds.apache.org/job/HBase-TRUNK/4622/])
HBASE-9773 Master aborted when hbck asked the master to assign a region that
was already online (jxiang: rev 1532633)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796447#comment-13796447
]
Hadoop QA commented on HBASE-9754:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608649/9754-rp-0.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7559//console
This message is automatically generated.
> Consider eliminating threadlocals from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-rp-0.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9779) IntegrationTestLoadAndVerify fails deleting IntegrationTestLoadAndVerify table
[
https://issues.apache.org/jira/browse/HBASE-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-9779:
-
Attachment: 9779part.txt
Fix so we don't retry if table doesn't exist.
If Elliott is reading, this failed the #36, #40 and half if not all of build
#38. Let me make it critical.
> IntegrationTestLoadAndVerify fails deleting IntegrationTestLoadAndVerify
> table
> ---
>
> Key: HBASE-9779
> URL: https://issues.apache.org/jira/browse/HBASE-9779
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: stack
> Attachments: 9779part.txt
>
>
> As part of the test, we want to delete the created table to restore cluster
> state. Interestingly we can disable the table successfully but then
> immediately after we fail the delete because we cannot get the table
> descriptor -- getting the file descriptor is used to test if table is present.
> The test for getDescriptor is kinda broke because it throws base IOE which
> causes clients to retry over and over again as though the descriptor was
> going to come back.
> This bug is kinda ugly because in at least one case it caused our
> long-running hbase-it suite run to fail so would be good to fix.
> Here is sample from a test run:
> {code}
> Disabling table IntegrationTestLoadAndVerify 2013-10-11 18:27:53,485 INFO
> [main] client.HBaseAdmin: Started disable of IntegrationTestLoadAndVerify
> 2013-10-11 18:27:53,526 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
> watcher=catalogtracker-on-hconnection-0x5a7e666f
> 2013-10-11 18:27:53,527 INFO [main] zookeeper.RecoverableZooKeeper: Process
> identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
> ensemble=a1805.halxg.cloudera.com:2181
> 2013-10-11 18:27:53,527 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Opening socket connection to server
> a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2013-10-11 18:27:53,527 DEBUG [main] catalog.CatalogTracker: Starting catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4ace08a5
> 2013-10-11 18:27:53,529 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket
> connection established to a1805.halxg.cloudera.com/10.20.200.105:2181,
> initiating session
> 2013-10-11 18:27:53,539 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Session establishment complete on server
> a1805.halxg.cloudera.com/10.20.200.105:2181, sessionid = 0x1412d47f53a5c70,
> negotiated timeout = 4
> 2013-10-11 18:27:53,602 DEBUG [main] catalog.CatalogTracker: Stopping catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4ace08a5
> 2013-10-11 18:27:53,662 INFO [main] zookeeper.ZooKeeper: Session:
> 0x1412d47f53a5c70 closed
> 2013-10-11 18:27:53,662 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> .2013-10-11 18:27:54,666 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
> watcher=catalogtracker-on-hconnection-0x5a7e666f
> 2013-10-11 18:27:54,667 INFO [main] zookeeper.RecoverableZooKeeper: Process
> identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
> ensemble=a1805.halxg.cloudera.com:2181
> 2013-10-11 18:27:54,667 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Opening socket connection to server
> a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2013-10-11 18:27:54,667 DEBUG [main] catalog.CatalogTracker: Starting catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@692c0c5d
> 2013-10-11 18:27:54,667 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket
> connection established to a1805.halxg.cloudera.com/10.20.200.105:2181,
> initiating session
> 2013-10-11 18:27:54,696 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Session establishment complete on server
> a1805.halxg.cloudera.com/10.20.200.105:2181, sessionid = 0x1412d47f53a5c71,
> negotiated timeout = 4
> 2013-10-11 18:27:54,821 DEBUG [main] catalog.CatalogTracker: Stopping catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@692c0c5d
> 2013-10-11 18:27:54,871 INFO [main] zookeeper.ZooKeeper: Session:
> 0x1412d47f53a5c71 closed
> 2013-10-11 18:27:54,871 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> .2013-10-11 18:27:55,890 INFO [main] zookeeper.ZooKeeper: Initiating client
> con
[jira] [Updated] (HBASE-9779) IntegrationTestLoadAndVerify fails deleting IntegrationTestLoadAndVerify table
[
https://issues.apache.org/jira/browse/HBASE-9779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-9779:
-
Priority: Critical (was: Major)
> IntegrationTestLoadAndVerify fails deleting IntegrationTestLoadAndVerify
> table
> ---
>
> Key: HBASE-9779
> URL: https://issues.apache.org/jira/browse/HBASE-9779
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 9779part.txt
>
>
> As part of the test, we want to delete the created table to restore cluster
> state. Interestingly we can disable the table successfully but then
> immediately after we fail the delete because we cannot get the table
> descriptor -- getting the file descriptor is used to test if table is present.
> The test for getDescriptor is kinda broke because it throws base IOE which
> causes clients to retry over and over again as though the descriptor was
> going to come back.
> This bug is kinda ugly because in at least one case it caused our
> long-running hbase-it suite run to fail so would be good to fix.
> Here is sample from a test run:
> {code}
> Disabling table IntegrationTestLoadAndVerify 2013-10-11 18:27:53,485 INFO
> [main] client.HBaseAdmin: Started disable of IntegrationTestLoadAndVerify
> 2013-10-11 18:27:53,526 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
> watcher=catalogtracker-on-hconnection-0x5a7e666f
> 2013-10-11 18:27:53,527 INFO [main] zookeeper.RecoverableZooKeeper: Process
> identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
> ensemble=a1805.halxg.cloudera.com:2181
> 2013-10-11 18:27:53,527 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Opening socket connection to server
> a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2013-10-11 18:27:53,527 DEBUG [main] catalog.CatalogTracker: Starting catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4ace08a5
> 2013-10-11 18:27:53,529 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket
> connection established to a1805.halxg.cloudera.com/10.20.200.105:2181,
> initiating session
> 2013-10-11 18:27:53,539 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Session establishment complete on server
> a1805.halxg.cloudera.com/10.20.200.105:2181, sessionid = 0x1412d47f53a5c70,
> negotiated timeout = 4
> 2013-10-11 18:27:53,602 DEBUG [main] catalog.CatalogTracker: Stopping catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4ace08a5
> 2013-10-11 18:27:53,662 INFO [main] zookeeper.ZooKeeper: Session:
> 0x1412d47f53a5c70 closed
> 2013-10-11 18:27:53,662 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> .2013-10-11 18:27:54,666 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
> watcher=catalogtracker-on-hconnection-0x5a7e666f
> 2013-10-11 18:27:54,667 INFO [main] zookeeper.RecoverableZooKeeper: Process
> identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
> ensemble=a1805.halxg.cloudera.com:2181
> 2013-10-11 18:27:54,667 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Opening socket connection to server
> a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2013-10-11 18:27:54,667 DEBUG [main] catalog.CatalogTracker: Starting catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@692c0c5d
> 2013-10-11 18:27:54,667 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn: Socket
> connection established to a1805.halxg.cloudera.com/10.20.200.105:2181,
> initiating session
> 2013-10-11 18:27:54,696 INFO
> [main-SendThread(a1805.halxg.cloudera.com:2181)] zookeeper.ClientCnxn:
> Session establishment complete on server
> a1805.halxg.cloudera.com/10.20.200.105:2181, sessionid = 0x1412d47f53a5c71,
> negotiated timeout = 4
> 2013-10-11 18:27:54,821 DEBUG [main] catalog.CatalogTracker: Stopping catalog
> tracker org.apache.hadoop.hbase.catalog.CatalogTracker@692c0c5d
> 2013-10-11 18:27:54,871 INFO [main] zookeeper.ZooKeeper: Session:
> 0x1412d47f53a5c71 closed
> 2013-10-11 18:27:54,871 INFO [main-EventThread] zookeeper.ClientCnxn:
> EventThread shut down
> .2013-10-11 18:27:55,890 INFO [main] zookeeper.ZooKeeper: Initiating client
> connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
> watcher=catalogtracker-on-hconnection-0
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796445#comment-13796445
]
Lars Hofhansl commented on HBASE-9754:
--
I think in the *PolicyObserver.preStoreScannerOpen, we want the current MVCC
readpoint, rather than the store smallest readpoint.
I am also a bit concerned about passing 0 as readPt in some places, as that
makes everything visible. And for example it avoids setting the memstoreTs of
KV to 0 even when there are no old scanners open that could see that KV.
(In fact these were the consideration that made me think a correct patch for
this is hard)
> Consider eliminating threadlocals from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-rp-0.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9779) IntegrationTestLoadAndVerify fails deleting IntegrationTestLoadAndVerify table
stack created HBASE-9779:
Summary: IntegrationTestLoadAndVerify fails deleting
IntegrationTestLoadAndVerify table
Key: HBASE-9779
URL: https://issues.apache.org/jira/browse/HBASE-9779
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: stack
Assignee: stack
As part of the test, we want to delete the created table to restore cluster
state. Interestingly we can disable the table successfully but then
immediately after we fail the delete because we cannot get the table descriptor
-- getting the file descriptor is used to test if table is present.
The test for getDescriptor is kinda broke because it throws base IOE which
causes clients to retry over and over again as though the descriptor was going
to come back.
This bug is kinda ugly because in at least one case it caused our long-running
hbase-it suite run to fail so would be good to fix.
Here is sample from a test run:
{code}
Disabling table IntegrationTestLoadAndVerify 2013-10-11 18:27:53,485 INFO
[main] client.HBaseAdmin: Started disable of IntegrationTestLoadAndVerify
2013-10-11 18:27:53,526 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
watcher=catalogtracker-on-hconnection-0x5a7e666f
2013-10-11 18:27:53,527 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
ensemble=a1805.halxg.cloudera.com:2181
2013-10-11 18:27:53,527 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
using SASL (unknown error)
2013-10-11 18:27:53,527 DEBUG [main] catalog.CatalogTracker: Starting catalog
tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4ace08a5
2013-10-11 18:27:53,529 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Socket connection established to
a1805.halxg.cloudera.com/10.20.200.105:2181, initiating session
2013-10-11 18:27:53,539 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
a1805.halxg.cloudera.com/10.20.200.105:2181, sessionid = 0x1412d47f53a5c70,
negotiated timeout = 4
2013-10-11 18:27:53,602 DEBUG [main] catalog.CatalogTracker: Stopping catalog
tracker org.apache.hadoop.hbase.catalog.CatalogTracker@4ace08a5
2013-10-11 18:27:53,662 INFO [main] zookeeper.ZooKeeper: Session:
0x1412d47f53a5c70 closed
2013-10-11 18:27:53,662 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
.2013-10-11 18:27:54,666 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
watcher=catalogtracker-on-hconnection-0x5a7e666f
2013-10-11 18:27:54,667 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
ensemble=a1805.halxg.cloudera.com:2181
2013-10-11 18:27:54,667 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
using SASL (unknown error)
2013-10-11 18:27:54,667 DEBUG [main] catalog.CatalogTracker: Starting catalog
tracker org.apache.hadoop.hbase.catalog.CatalogTracker@692c0c5d
2013-10-11 18:27:54,667 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Socket connection established to
a1805.halxg.cloudera.com/10.20.200.105:2181, initiating session
2013-10-11 18:27:54,696 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Session establishment complete on server
a1805.halxg.cloudera.com/10.20.200.105:2181, sessionid = 0x1412d47f53a5c71,
negotiated timeout = 4
2013-10-11 18:27:54,821 DEBUG [main] catalog.CatalogTracker: Stopping catalog
tracker org.apache.hadoop.hbase.catalog.CatalogTracker@692c0c5d
2013-10-11 18:27:54,871 INFO [main] zookeeper.ZooKeeper: Session:
0x1412d47f53a5c71 closed
2013-10-11 18:27:54,871 INFO [main-EventThread] zookeeper.ClientCnxn:
EventThread shut down
.2013-10-11 18:27:55,890 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=a1805.halxg.cloudera.com:2181 sessionTimeout=9
watcher=catalogtracker-on-hconnection-0x5a7e666f
2013-10-11 18:27:55,891 INFO [main] zookeeper.RecoverableZooKeeper: Process
identifier=catalogtracker-on-hconnection-0x5a7e666f connecting to ZooKeeper
ensemble=a1805.halxg.cloudera.com:2181
2013-10-11 18:27:55,891 INFO [main-SendThread(a1805.halxg.cloudera.com:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
a1805.halxg.cloudera.com/10.20.200.105:2181. Will not attempt to authenticate
using SASL (unknown error)
2013
[jira] [Updated] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[ https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9778: - Description: The issue of slow seeking in ExplicitColumnTracker was brought up by [~vrodionov] on the dev list. My idea here is to avoid the seeking if we know that there aren't many versions to skip. How do we know? We'll use the column family's VERSIONS setting as a hint. If VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call SKIP repeatedly. HBASE-9769 has some initial number for this approach: Interestingly it depends on which column(s) is (are) selected. Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, everything filtered at the server with a ValueFilter. Everything measured in seconds. Without patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.5|14.3|14.6|11.1|20.3| With patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.4|8.9|9.9|6.4|10.0| Variation here was +- 0.2s. So with this patch scanning is 2x faster than without in some cases, and never slower. No special hint needed, beyond declaring VERSIONS correctly. was: The issue of slow seeking in ExplicitColumnTracker was brought up by [~vrodionov] on the dev list. My idea here is to avoid the seeking if we know that there aren't many rows to skip. How do we know? We'll use the column family's VERSIONS setting as a hint. If VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call SKIP repeatedly. HBASE-9769 has some initial number for this approach: Interestingly it depends on which column(s) is (are) selected. Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, everything filtered at the server with a ValueFilter. Everything measured in seconds. Without patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.5|14.3|14.6|11.1|20.3| With patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.4|8.9|9.9|6.4|10.0| Variation here was +- 0.2s. So with this patch scanning is 2x faster than without in some cases, and never slower. No special hint needed, beyond declaring VERSIONS correctly. > Avoid seeking to next column in ExplicitColumnTracker when possible > --- > > Key: HBASE-9778 > URL: https://issues.apache.org/jira/browse/HBASE-9778 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9778-0.94.txt, 9778-trunk.txt > > > The issue of slow seeking in ExplicitColumnTracker was brought up by > [~vrodionov] on the dev list. > My idea here is to avoid the seeking if we know that there aren't many > versions to skip. > How do we know? We'll use the column family's VERSIONS setting as a hint. If > VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call > SKIP repeatedly. > HBASE-9769 has some initial number for this approach: > Interestingly it depends on which column(s) is (are) selected. > Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, > everything filtered at the server with a ValueFilter. Everything measured in > seconds. > Without patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.5|14.3|14.6|11.1|20.3| > With patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.4|8.9|9.9|6.4|10.0| > Variation here was +- 0.2s. > So with this patch scanning is 2x faster than without in some cases, and > never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796436#comment-13796436 ] Lars Hofhansl commented on HBASE-9769: -- Nit: Use the Eclipse formatter from HBASE-5961. We use 2 spaces instead of a tab for indentation. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796435#comment-13796435 ] Lars Hofhansl commented on HBASE-9769: -- I created HBASE-9778 for my patch idea. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796434#comment-13796434 ] Lars Hofhansl commented on HBASE-9769: -- MAX_VERSIONS=1 (or a low number) can only be used to eliminate the NEXT_COL seek (as that is use to seek past versions of the same column). It does not indicate anything about the number of columns in a row, and hence we know nothing about whether SEEK_NEXT_ROW or a series of SKIPs is better. We need both, I think. (MAX_VERSIONS is a hint in the sense that there temporarily can be more versions in the memstore and/or distributed over various HFiles, only after a major compaction will the number of versions actually be <= MAX_VERSIONS.) > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796433#comment-13796433 ] Vladimir Rodionov commented on HBASE-9769: -- Lars, our patches are independent. I think they need to be merged into one, or you better create new JIRA for *do-not-seek-next-col* thing. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[
https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796431#comment-13796431
]
Vladimir Rodionov commented on HBASE-9769:
--
To activate this feature (hint):
{code}
Scan scan = ...
scan.setAttribute(Scan.SCAN_NARROW_ROWS, "true".getBytes());
{code}
OK. I think I will replace SCAN_NARROW_ROWS with HINT_NARROW_ROWS.
> Improve performance of a Scanner with explicit column list when rows are
> small/medium size
> --
>
> Key: HBASE-9769
> URL: https://issues.apache.org/jira/browse/HBASE-9769
> Project: HBase
> Issue Type: Improvement
> Components: Scanners
>Affects Versions: 0.98.0, 0.94.12, 0.96.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt,
> 9769-0.94-sample.txt, 9769-94.txt
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796430#comment-13796430 ] Vladimir Rodionov commented on HBASE-9769: -- It contains check MAX_VERSIONS = 1 suggested by Lars (not sure if it is really a hint?). Lars version gives improvements as well, but it relies on default hint of MAX_VERSIONS and is slower I think. I completely eliminated ExplicitColumnTracker from the code path. The more columns in a scan the more is going to be performance difference, I think (again). > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796426#comment-13796426 ] Vladimir Rodionov commented on HBASE-9769: -- It contains check MAX_VERSIONS = 1 suggested by Lars (not sure if it is really a hint?). > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[ https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9778: - Attachment: 9778-trunk.txt And a trunk version for HadoopQA > Avoid seeking to next column in ExplicitColumnTracker when possible > --- > > Key: HBASE-9778 > URL: https://issues.apache.org/jira/browse/HBASE-9778 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9778-0.94.txt, 9778-trunk.txt > > > The issue of slow seeking in ExplicitColumnTracker was brought up by > [~vrodionov] on the dev list. > My idea here is to avoid the seeking if we know that there aren't many rows > to skip. > How do we know? We'll use the column family's VERSIONS setting as a hint. If > VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call > SKIP repeatedly. > HBASE-9769 has some initial number for this approach: > Interestingly it depends on which column(s) is (are) selected. > Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, > everything filtered at the server with a ValueFilter. Everything measured in > seconds. > Without patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.5|14.3|14.6|11.1|20.3| > With patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.4|8.9|9.9|6.4|10.0| > Variation here was +- 0.2s. > So with this patch scanning is 2x faster than without in some cases, and > never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[ https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9778: - Attachment: 9778-0.94.txt 0.94 patch. > Avoid seeking to next column in ExplicitColumnTracker when possible > --- > > Key: HBASE-9778 > URL: https://issues.apache.org/jira/browse/HBASE-9778 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9778-0.94.txt, 9778-trunk.txt > > > The issue of slow seeking in ExplicitColumnTracker was brought up by > [~vrodionov] on the dev list. > My idea here is to avoid the seeking if we know that there aren't many rows > to skip. > How do we know? We'll use the column family's VERSIONS setting as a hint. If > VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call > SKIP repeatedly. > HBASE-9769 has some initial number for this approach: > Interestingly it depends on which column(s) is (are) selected. > Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, > everything filtered at the server with a ValueFilter. Everything measured in > seconds. > Without patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.5|14.3|14.6|11.1|20.3| > With patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.4|8.9|9.9|6.4|10.0| > Variation here was +- 0.2s. > So with this patch scanning is 2x faster than without in some cases, and > never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[ https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9778: - Status: Patch Available (was: Open) > Avoid seeking to next column in ExplicitColumnTracker when possible > --- > > Key: HBASE-9778 > URL: https://issues.apache.org/jira/browse/HBASE-9778 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9778-0.94.txt, 9778-trunk.txt > > > The issue of slow seeking in ExplicitColumnTracker was brought up by > [~vrodionov] on the dev list. > My idea here is to avoid the seeking if we know that there aren't many rows > to skip. > How do we know? We'll use the column family's VERSIONS setting as a hint. If > VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call > SKIP repeatedly. > HBASE-9769 has some initial number for this approach: > Interestingly it depends on which column(s) is (are) selected. > Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, > everything filtered at the server with a ValueFilter. Everything measured in > seconds. > Without patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.5|14.3|14.6|11.1|20.3| > With patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.4|8.9|9.9|6.4|10.0| > Variation here was +- 0.2s. > So with this patch scanning is 2x faster than without in some cases, and > never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
[ https://issues.apache.org/jira/browse/HBASE-9778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796423#comment-13796423 ] Lars Hofhansl commented on HBASE-9778: -- Note that this avoid the NEXT_COL but not the NEXT_ROW seeks. Here we have nothing to go by to know whether a row is going to be large (many columns) or not, and hence we'd need to hint the scan explicitly somehow and/or use a filter as Vladimir suggests in HBASE-9769. > Avoid seeking to next column in ExplicitColumnTracker when possible > --- > > Key: HBASE-9778 > URL: https://issues.apache.org/jira/browse/HBASE-9778 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.98.0, 0.94.13, 0.96.1 > > > The issue of slow seeking in ExplicitColumnTracker was brought up by > [~vrodionov] on the dev list. > My idea here is to avoid the seeking if we know that there aren't many rows > to skip. > How do we know? We'll use the column family's VERSIONS setting as a hint. If > VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call > SKIP repeatedly. > HBASE-9769 has some initial number for this approach: > Interestingly it depends on which column(s) is (are) selected. > Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, > everything filtered at the server with a ValueFilter. Everything measured in > seconds. > Without patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.5|14.3|14.6|11.1|20.3| > With patch: > ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| > |6.4|8.4|8.9|9.9|6.4|10.0| > Variation here was +- 0.2s. > So with this patch scanning is 2x faster than without in some cases, and > never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-9769: - Attachment: 9769-94.txt Initial version of a patch > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt, 9769-94.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9778) Avoid seeking to next column in ExplicitColumnTracker when possible
Lars Hofhansl created HBASE-9778: Summary: Avoid seeking to next column in ExplicitColumnTracker when possible Key: HBASE-9778 URL: https://issues.apache.org/jira/browse/HBASE-9778 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.98.0, 0.94.13, 0.96.1 The issue of slow seeking in ExplicitColumnTracker was brought up by [~vrodionov] on the dev list. My idea here is to avoid the seeking if we know that there aren't many rows to skip. How do we know? We'll use the column family's VERSIONS setting as a hint. If VERSIONS is set to 1 (or maybe some value < 10) we'll avoid the seek and call SKIP repeatedly. HBASE-9769 has some initial number for this approach: Interestingly it depends on which column(s) is (are) selected. Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1, everything filtered at the server with a ValueFilter. Everything measured in seconds. Without patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.5|14.3|14.6|11.1|20.3| With patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.4|8.9|9.9|6.4|10.0| Variation here was +- 0.2s. So with this patch scanning is 2x faster than without in some cases, and never slower. No special hint needed, beyond declaring VERSIONS correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796417#comment-13796417 ] Lars Hofhansl commented on HBASE-9769: -- bq. Lars, HTable can have small number of versions and large number of column qualifiers or large values (say 100K). That is true. Seeking to the next column is not a good idea, though, if we know there are not going to be many versions to skip. So the suggested patch here will not be slower than before, and it will improve performance in many cases. As the size of a KV approaches the HFile blocksize (64k by default), SKIP and SEEK_NEXT_COL should become equivalent in performance (in both cases we'll need to find the KV in the next block). As I said, this does not eliminate the NEXT_ROW seeking. I fear the filter approach will lead to issues when there are already filters configured on the scan. You'd have to convert this to a FilterList while keeping all the semantics and performance characteristics. I think it might be best to ship your Filter and document its use. I'll file a separate issue for my patch. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796414#comment-13796414
]
stack commented on HBASE-9776:
--
Thanks [~jeffreyz]
Looking at the delete table code, it is doing a getDescription to test
existence. Rather than return a TableNotFoundException which is a DoNotRetry
exception, the code is throwing an IOE which gets retried 35 times in a matter
of seconds. Could do with some improvement. Let me look at it.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796411#comment-13796411 ] Lars Hofhansl commented on HBASE-9769: -- Note that Vladimir's small row hint still can be used to eliminate the NEXT_ROW seek. Maybe, again, it is prudent to split this in two issues. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9769: - Attachment: 9769-0.94-sample2.txt Even simpler. Same perf numbers. (The previous patch was actually not correct when there are actually multiple versions). > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, > 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796408#comment-13796408 ] Lars Hofhansl commented on HBASE-9769: -- Interestingly it depends on which column(s) is (are) selected. Some numbers: 4m rows, 5 cols each, 1 cf, 10 bytes values, VERSIONS=1. Everything measured in seconds. Without patch: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.5|14.3|14.6|11.1|20.3| With patch sample1: ||Wildcard||Col 1||Col 2||Col 4||Col 5||Col 2+4|| |6.4|8.4|8.9|9.9|6.4|10.0| Variation here was +- 0.2s. So with this patch scanning is 2x faster than without in some cases, and never slower. No special hint needed, beyond declaring VERSIONS correctly. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script
[
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796400#comment-13796400
]
ramkrishna.s.vasudevan commented on HBASE-9689:
---
[~apurtell]
bq.There is no way to have a backwards compatible put command that does not
take an attributes hash, just a timestamp
You can do that.
{code}
put 't1', 'r1', 'c1', 'value', {TIMESTAMP=>100}
{code}
What i meant was Timstamp cannot be specified as per the older way
{code}
put 't1','r1','value',ts
{code}
The problem is put accepts a set of parameters. So if we avoid the ts
parameter and specify the ATTRIBUTE instead the timestamp is taken as
'mykeymyvalue'.
> Support using OperationAttributes through shell script
> --
>
> Key: HBASE-9689
> URL: https://issues.apache.org/jira/browse/HBASE-9689
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.98.0
>
> Attachments: HBASE-9689_1.patch, HBASE-9689_2.patch, HBASE-9689.patch
>
>
> Currently the ruby scripts for Put does not support setting of Operation
> Attributes through shell.
> It may be useful in some cases and also for testing. And that would give a
> full fledged support using shell.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9754:
--
Attachment: 9754-rp-0.txt
Rebased patch.
> Consider eliminating threadlocals from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-rp-0.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9754:
--
Attachment: (was: 9754-rp-0.txt)
> Consider eliminating threadlocals from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-rp-0.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796384#comment-13796384 ] stack commented on HBASE-5487: -- @enis List makes for pretty good set of requirements. We used to talk 100k regions but folks are long past that now so we are behind the curve (Flurry are >250k IIRC) and we may want to tend away from a few large regions and more toward many small regions if we can get AM to perform (advantages: smaller compression runs, easier to free up WALs, etc) > Generic framework for Master-coordinated tasks > -- > > Key: HBASE-5487 > URL: https://issues.apache.org/jira/browse/HBASE-5487 > Project: HBase > Issue Type: New Feature > Components: master, regionserver, Zookeeper >Affects Versions: 0.94.0 >Reporter: Mubarak Seyed >Priority: Critical > Attachments: Region management in Master5.docx, Region management in > Master.pdf > > > Need a framework to execute master-coordinated tasks in a fault-tolerant > manner. > Master-coordinated tasks such as online-scheme change and delete-range > (deleting region(s) based on start/end key) can make use of this framework. > The advantages of framework are > 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for > master-coordinated tasks > 2. Ability to abstract the common functions across Master -> ZK and RS -> ZK > 3. Easy to plugin new master-coordinated tasks without adding code to core > components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9771) [WebUI] Humanize store and blockcache statistics on RS
[ https://issues.apache.org/jira/browse/HBASE-9771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796380#comment-13796380 ] Hudson commented on HBASE-9771: --- SUCCESS: Integrated in hbase-0.96 #142 (See [https://builds.apache.org/job/hbase-0.96/142/]) HBASE-9771 [WebUI] Humanize store and blockcache statistics on RS (ndimiduk: rev 1532599) * /hbase/branches/0.96/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java * /hbase/branches/0.96/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon > [WebUI] Humanize store and blockcache statistics on RS > -- > > Key: HBASE-9771 > URL: https://issues.apache.org/jira/browse/HBASE-9771 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Fix For: 0.98.0, 0.96.1 > > Attachments: HBASE-9771.00.patch, HBASE-9771.01.patch, > HBASE-9771.02.patch > > > Some statistics reported on webui don't include hints about what the unit is, > leaving people guessing about what they mean. Clean them up. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-5487: Attachment: Region management in Master5.docx attaching preliminary version... still need to iron out all operation details, where step-by-step operation starts many things may be wrong, but it should be good until then. Sorry Mac craps out when printing this as PDF due to mix of page orientations, let me try to find some better format for final version... > Generic framework for Master-coordinated tasks > -- > > Key: HBASE-5487 > URL: https://issues.apache.org/jira/browse/HBASE-5487 > Project: HBase > Issue Type: New Feature > Components: master, regionserver, Zookeeper >Affects Versions: 0.94.0 >Reporter: Mubarak Seyed >Priority: Critical > Attachments: Region management in Master5.docx, Region management in > Master.pdf > > > Need a framework to execute master-coordinated tasks in a fault-tolerant > manner. > Master-coordinated tasks such as online-scheme change and delete-range > (deleting region(s) based on start/end key) can make use of this framework. > The advantages of framework are > 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for > master-coordinated tasks > 2. Ability to abstract the common functions across Master -> ZK and RS -> ZK > 3. Easy to plugin new master-coordinated tasks without adding code to core > components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796365#comment-13796365
]
stack commented on HBASE-9773:
--
[~jxiang] makes sense. I like the test.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796362#comment-13796362 ] Enis Soztutar commented on HBASE-5487: -- I also started a document some time ago, but never got to finish it to the level of details I would like. However, I think we can agree on the design goals section which I augmented from the discussion so far: - Robust implementation - Compressive test coverage by mocking server and region assignment states (unit testable without MiniCluster and CM stuff) - Bulk region operations - Region operations should be isolated from server operations (AM vs SSH, log splitting), and table operations (disabling / disabled table, schema changes, etc) and cluster shutdown. AM and SSH should NEVER know about table state (disable/disabling). Server liveness checks can only be done as an optimization (servers can fail after the check is done) - There should be one source of truth - Should be compatible with master failover, and concurrent region operations(split, RS failover, balancer, etc) - AM should guarantee that a region can be hosted by a single region server at any given time - AM should be understandable by simple human beings like myself - Actions for AM should be logged (possibly separately). We would like to be able to construct the history for the regions from logs or some persisted state. - Assignment should be performant and parallelizable. We should target handling millions of regions and thousands of servers. A single region assignment should complete under 1 sec. (1PB data with 1 GB regions = 1M regions) - No master abort when a region’s state cannot be determined. This results in support cases where master cannot start, and without master things become even worse. We should “quarantine” the regions if needed absolutely. > Generic framework for Master-coordinated tasks > -- > > Key: HBASE-5487 > URL: https://issues.apache.org/jira/browse/HBASE-5487 > Project: HBase > Issue Type: New Feature > Components: master, regionserver, Zookeeper >Affects Versions: 0.94.0 >Reporter: Mubarak Seyed >Priority: Critical > Attachments: Region management in Master.pdf > > > Need a framework to execute master-coordinated tasks in a fault-tolerant > manner. > Master-coordinated tasks such as online-scheme change and delete-range > (deleting region(s) based on start/end key) can make use of this framework. > The advantages of framework are > 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for > master-coordinated tasks > 2. Ability to abstract the common functions across Master -> ZK and RS -> ZK > 3. Easy to plugin new master-coordinated tasks without adding code to core > components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796357#comment-13796357
]
chunhui shen commented on HBASE-9754:
-
Patch looks good.
Agree with eliminating the threadlocals
> Consider eliminating threadlocals from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-rp-0.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9773:
---
Resolution: Fixed
Fix Version/s: 0.96.1
0.98.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Integrated into trunk and 0.96. Thanks Devaraj for finding the issue. Thanks
Stack, Ted for reviewing the patch.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9769: - Attachment: 9769-0.94-sample1.txt With VERSIONS=1 only. I think I like this better. User declares to only be interested in 1 version, in that case we use SKIP and INCLUDE and not column seeking. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796341#comment-13796341
]
Jimmy Xiang edited comment on HBASE-9773 at 10/16/13 2:43 AM:
--
[~saint@gmail.com], if it is null, the state is managed by the caller so we
leave it to the caller. If the transitionInZK is not set, the caller is just to
do the best to close the region just in case to avoid double assignment. So we
don't check the actual state here.
was (Author: jxiang):
[~saint@gmail.com], if it is null, the state is managed by the caller so we
leave it to the caller.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796348#comment-13796348 ] Vladimir Rodionov commented on HBASE-9769: -- Lars, HTable can have small number of versions and large number of column qualifiers or large values (say 100K). > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796346#comment-13796346 ] Lars Hofhansl edited comment on HBASE-9769 at 10/16/13 2:38 AM: Here's a *sample* of what I mean. 10 is hardcoded. Alternatively we could only do this when VERSIONS=1, so this way the user would naturally provide a hint about how many version (s)he expects to keep and we use that hint. was (Author: lhofhansl): Here's a *sample* of what I mean. 10 is hardcoded. Alternatively we could only do this when VERSIONS=1, so way the use would naturally provide a hint about how many version (s)he expects to keep and we use that hint. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9771) [WebUI] Humanize store and blockcache statistics on RS
[ https://issues.apache.org/jira/browse/HBASE-9771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796347#comment-13796347 ] Hudson commented on HBASE-9771: --- SUCCESS: Integrated in HBase-TRUNK #4621 (See [https://builds.apache.org/job/HBase-TRUNK/4621/]) HBASE-9771 [WebUI] Humanize store and blockcache statistics on RS (ndimiduk: rev 1532598) * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon > [WebUI] Humanize store and blockcache statistics on RS > -- > > Key: HBASE-9771 > URL: https://issues.apache.org/jira/browse/HBASE-9771 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Fix For: 0.98.0, 0.96.1 > > Attachments: HBASE-9771.00.patch, HBASE-9771.01.patch, > HBASE-9771.02.patch > > > Some statistics reported on webui don't include hints about what the unit is, > leaving people guessing about what they mean. Clean them up. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9769: - Attachment: 9769-0.94-sample.txt Here's a *sample* of what I mean. 10 is hardcoded. Alternatively we could only do this when VERSIONS=1, so way the use would naturally provide a hint about how many version (s)he expects to keep and we use that hint. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 9769-0.94-sample.txt > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796345#comment-13796345 ] Lars Hofhansl commented on HBASE-9769: -- Another idea I had was to make use of the column family's VERSIONS setting. If it is "small" use INCLUDE and SKIP in the ExplicitColumnTracker, otherwise use INCLUDE_AND_SEEK_NEXT_COL and SEEK_NEXT_COL. In my tests this yields a nice improvement bringing ExplicitColumnTracker on par with ScanWildcardColumnTracker. For now I defined "small" as 10, but that needs to be tested more. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796341#comment-13796341
]
Jimmy Xiang commented on HBASE-9773:
[~saint@gmail.com], if it is null, the state is managed by the caller so we
leave it to the caller.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796337#comment-13796337
]
Hadoop QA commented on HBASE-9776:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608617/hbase-9776.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7558//console
This message is automatically generated.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796331#comment-13796331 ] Vladimir Rodionov commented on HBASE-9769: -- Its a server-side filter and is not meant to be exposed to HBase client. The reason: it has only list of qualifiers - no columns. It is instantiated in StoreScanner. If Scan has already filter, the new FilterList is created with MUST_PASS_ALL operator. First goes ExplicitColumnsFilter then existing filter. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796311#comment-13796311 ] chunhui shen edited comment on HBASE-9769 at 10/16/13 1:45 AM: --- Move the logic of above patch to Scan class, is it also OK? It means adding the ExplicitColumnsFilter in Scan.java when setting the attribute "SCAN-SMALL-ROWS" In addition, I worry about the data correctness if Scan already has complex filters. was (Author: zjushch): Move the logic of above patch to Scan class, is it also OK? It means adding the ExplicitColumnsFilter in Scan.java when setting the attribute "SCAN-SMALL-ROWS" In addition, I'm worry the data correctness if Scan already has complex filters. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796311#comment-13796311 ] chunhui shen commented on HBASE-9769: - Move the logic of above patch to Scan class, is it also OK? It means adding the ExplicitColumnsFilter in Scan.java when setting the attribute "SCAN-SMALL-ROWS" In addition, I'm worry the data correctness if Scan already has complex filters. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796305#comment-13796305
]
Jeffrey Zhong commented on HBASE-9776:
--
[~saint@gmail.com] I'm afraid you hit a different issue. The above stack
trace you posted seems we have a half done deletion before and subsequent
retries all failed because of that. Since delete/disable/create table
operations aren't idempotent, executeCallable on these table operations is
problematic. I guess we need a FATE like model for table operations.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9775) Client write path perf issues
[
https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796294#comment-13796294
]
Elliott Clark commented on HBASE-9775:
--
YCSB with 33 nodes fails completely. with:
{code}
com.yahoo.ycsb.DBException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
8608 actions: IOException: 8608 times,
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:113)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 8608 actions: IOException: 8608 times,
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:185)
at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:169)
at
org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:782)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:934)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1193)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:108)
... 2 more
{code}
running:
{code}
#!/bin/bash
OPERTATIONS=2000
RECORDS=$(($OPERTATIONS*10))
echo "disable 'usertable'; drop 'usertable'" | hbase shell
echo "create 'usertable',{NAME => 'd', VERSIONS => 1,COMPRESSION => 'lz4',
BLOCKCACHE => true}" | hbase shell
echo "balancer" | hbase shell
sleep 10
cd /home/eclark/RC5/ycsb/
bin/ycsb load hbase -P workloads/workloada -p columnfamily=d -s -threads 32 -p
recordcount=$RECORDS 2> /tmp/ycsb-workload-a-load-output.log
{code}
I've also seen the same stack and exception when trying to run ITBLL on this
cluster.
> Client write path perf issues
> -
>
> Key: HBASE-9775
> URL: https://issues.apache.org/jira/browse/HBASE-9775
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.96.0
>Reporter: Elliott Clark
>Priority: Critical
> Attachments: Charts Search Cloudera Manager.png, short_ycsb.png
>
>
> Testing on larger clusters has not had the desired throughput increases.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9775) Client write path perf issues
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-9775: - Summary: Client write path perf issues (was: Client write path scales very badly with more servers) > Client write path perf issues > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Priority: Critical > Attachments: Charts Search Cloudera Manager.png, short_ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9753) Excessive readpoint checks in MemstoreScanner
[ https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796284#comment-13796284 ] Hudson commented on HBASE-9753: --- FAILURE: Integrated in hbase-0.96-hadoop2 #89 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/89/]) HBASE-9753 Excessive readpoint checks in MemstoreScanner (tedyu: rev 1532431) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java > Excessive readpoint checks in MemstoreScanner > - > > Key: HBASE-9753 > URL: https://issues.apache.org/jira/browse/HBASE-9753 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Ted Yu > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9753-0.94.txt, 9753-v1.txt > > > Brought up by [~vrodionov] on the mailing list. See also HBASE-9751. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796285#comment-13796285
]
Hudson commented on HBASE-9570:
---
FAILURE: Integrated in hbase-0.96-hadoop2 #89 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/89/])
HBASE-9570 With AccessDeniedException, HBase shell would be better to just
display the error message to be user friendly (Yang Wang) (ndimiduk: rev
1532398)
* /hbase/branches/0.96/hbase-shell/src/main/ruby/shell/commands.rb
> With AccessDeniedException, HBase shell would be better to just display the
> error message to be user friendly
> -
>
> Key: HBASE-9570
> URL: https://issues.apache.org/jira/browse/HBASE-9570
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Minor
> Fix For: 0.98.0, 0.96.1
>
> Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch,
> HBASE-9570-0.96.01.patch
>
>
> When access unauthorized resource like table, AccessDeniedException will be
> thrown. In HBase shell, the error message with stack trace will be displayed
> as follows. It would be better to just display the error message avoiding the
> stack trace to be user friendly.
> {noformat}
> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'u1' for scanner open on table t1
> at
> org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
> at
> org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
> at
> org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9753) Excessive readpoint checks in MemstoreScanner
[ https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796282#comment-13796282 ] Hudson commented on HBASE-9753: --- FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #795 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/795/]) HBASE-9753 Excessive readpoint checks in MemstoreScanner (tedyu: rev 1532432) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java > Excessive readpoint checks in MemstoreScanner > - > > Key: HBASE-9753 > URL: https://issues.apache.org/jira/browse/HBASE-9753 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Ted Yu > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9753-0.94.txt, 9753-v1.txt > > > Brought up by [~vrodionov] on the mailing list. See also HBASE-9751. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9766) HFileV3 - Optional tags write and read is not working as expected
[ https://issues.apache.org/jira/browse/HBASE-9766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796280#comment-13796280 ] Andrew Purtell commented on HBASE-9766: --- +1 on patch v2 > HFileV3 - Optional tags write and read is not working as expected > - > > Key: HBASE-9766 > URL: https://issues.apache.org/jira/browse/HBASE-9766 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.98.0 > > Attachments: HBASE-9766.patch, HBASE-9766_V2.patch > > > In the writer V3 includesTags always comes as true only and so writing tags > length always even when compaction selection says not to do so. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9777) Two consecutive RS crashes could lead to their SSH stepping on each other's toes and cause master abort
Devaraj Das created HBASE-9777:
--
Summary: Two consecutive RS crashes could lead to their SSH
stepping on each other's toes and cause master abort
Key: HBASE-9777
URL: https://issues.apache.org/jira/browse/HBASE-9777
Project: HBase
Issue Type: Bug
Reporter: Devaraj Das
Here is the sequence of events:
1. Master assigns regions to some server RS1. One particular region is
300d71b112325d43b99b6148ec7bc5b3
2. RS1 crashes
3. Master tries to bulk-reassign (this has retries as well) the regions to
other RSs. Let's say one of them is RS2.
{noformat}
2013-10-14 21:16:22,218 INFO
[hor13n02.gq1.ygridcore.net,6,1381784464025-GeneralBulkAssigner-0]
master.RegionStates: Transitioned {300d71b112325d43b99b6148ec7bc5b3
state=OFFLINE, ts=1381785382125, server=null} to
{300d71b112325d43b99b6148ec7bc5b3 state=PENDING_OPEN, ts=1381785382218,
server=hor13n04.gq1.ygridcore.net,60020,1381784772417}
{noformat}
4. RS2 crashes
5. The ServerShutdownHandler for RS2 is executed, and it tries to reassign the
regions.
{noformat}
2013-10-14 21:16:32,185 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
master.RegionStates: Found opening region {300d71b112325d43b99b6148ec7bc5b3
state=PENDING_OPEN, ts=1381785382218,
server=hor13n04.gq1.ygridcore.net,60020,1381784772417} to be reassigned by SSH
for hor13n04.gq1.ygridcore.net,60020,1381784772417
{noformat}
6. (5) succeeds. The region states are made OPEN.
7. The retry from (3) kicks in
{noformat}
2013-10-14 21:16:22,222 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-1]
master.GeneralBulkAssigner: Failed assigning 52 regions to server
hor13n04.gq1.ygridcore.net,60020,1381784772417, reassigning them
{noformat}
8. The retry finds some region state as OPEN, and the master aborts with the
stack trace:
{noformat}
2013-10-14 21:16:34,342 FATAL AM.-pool1-t46 master.HMaster: Unexpected state :
{300d71b112325d43b99b6148ec7bc5b3 state=OPEN, ts=1381785392864,
server=hor13n08.gq1.ygridcore.net,60020,1381785385596} .. Cannot transit it to
OFFLINE.
java.lang.IllegalStateException: Unexpected state :
{300d71b112325d43b99b6148ec7bc5b3 state=OPEN, ts=1381785392864,
server=hor13n08.gq1.ygridcore.net,60020,1381785385596}
.. Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:2074)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1855)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1449)
at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:45)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9777) Two consecutive RS crashes could lead to their SSH stepping on each other's toes and cause master abort
[
https://issues.apache.org/jira/browse/HBASE-9777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-9777:
---
Description:
Here is the sequence of events (with a version of 0.96 very close to RC5
version created on 10/11):
1. Master assigns regions to some server RS1. One particular region is
300d71b112325d43b99b6148ec7bc5b3
2. RS1 crashes
3. Master tries to bulk-reassign (this has retries as well) the regions to
other RSs. Let's say one of them is RS2.
{noformat}
2013-10-14 21:16:22,218 INFO
[hor13n02.gq1.ygridcore.net,6,1381784464025-GeneralBulkAssigner-0]
master.RegionStates: Transitioned {300d71b112325d43b99b6148ec7bc5b3
state=OFFLINE, ts=1381785382125, server=null} to
{300d71b112325d43b99b6148ec7bc5b3 state=PENDING_OPEN, ts=1381785382218,
server=hor13n04.gq1.ygridcore.net,60020,1381784772417}
{noformat}
4. RS2 crashes
5. The ServerShutdownHandler for RS2 is executed, and it tries to reassign the
regions.
{noformat}
2013-10-14 21:16:32,185 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
master.RegionStates: Found opening region {300d71b112325d43b99b6148ec7bc5b3
state=PENDING_OPEN, ts=1381785382218,
server=hor13n04.gq1.ygridcore.net,60020,1381784772417} to be reassigned by SSH
for hor13n04.gq1.ygridcore.net,60020,1381784772417
{noformat}
6. (5) succeeds. The region states are made OPEN.
7. The retry from (3) kicks in
{noformat}
2013-10-14 21:16:22,222 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-1]
master.GeneralBulkAssigner: Failed assigning 52 regions to server
hor13n04.gq1.ygridcore.net,60020,1381784772417, reassigning them
{noformat}
8. The retry finds some region state as OPEN, and the master aborts with the
stack trace:
{noformat}
2013-10-14 21:16:34,342 FATAL AM.-pool1-t46 master.HMaster: Unexpected state :
{300d71b112325d43b99b6148ec7bc5b3 state=OPEN, ts=1381785392864,
server=hor13n08.gq1.ygridcore.net,60020,1381785385596} .. Cannot transit it to
OFFLINE.
java.lang.IllegalStateException: Unexpected state :
{300d71b112325d43b99b6148ec7bc5b3 state=OPEN, ts=1381785392864,
server=hor13n08.gq1.ygridcore.net,60020,1381785385596}
.. Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:2074)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1855)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1449)
at org.apache.hadoop.hbase.master.AssignCallable.call(AssignCallable.java:45)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
was:
Here is the sequence of events:
1. Master assigns regions to some server RS1. One particular region is
300d71b112325d43b99b6148ec7bc5b3
2. RS1 crashes
3. Master tries to bulk-reassign (this has retries as well) the regions to
other RSs. Let's say one of them is RS2.
{noformat}
2013-10-14 21:16:22,218 INFO
[hor13n02.gq1.ygridcore.net,6,1381784464025-GeneralBulkAssigner-0]
master.RegionStates: Transitioned {300d71b112325d43b99b6148ec7bc5b3
state=OFFLINE, ts=1381785382125, server=null} to
{300d71b112325d43b99b6148ec7bc5b3 state=PENDING_OPEN, ts=1381785382218,
server=hor13n04.gq1.ygridcore.net,60020,1381784772417}
{noformat}
4. RS2 crashes
5. The ServerShutdownHandler for RS2 is executed, and it tries to reassign the
regions.
{noformat}
2013-10-14 21:16:32,185 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
master.RegionStates: Found opening region {300d71b112325d43b99b6148ec7bc5b3
state=PENDING_OPEN, ts=1381785382218,
server=hor13n04.gq1.ygridcore.net,60020,1381784772417} to be reassigned by SSH
for hor13n04.gq1.ygridcore.net,60020,1381784772417
{noformat}
6. (5) succeeds. The region states are made OPEN.
7. The retry from (3) kicks in
{noformat}
2013-10-14 21:16:22,222 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-1]
master.GeneralBulkAssigner: Failed assigning 52 regions to server
hor13n04.gq1.ygridcore.net,60020,1381784772417, reassigning them
{noformat}
8. The retry finds some region state as OPEN, and the master aborts with the
stack trace:
{noformat}
2013-10-14 21:16:34,342 FATAL AM.-pool1-t46 master.HMaster: Unexpected state :
{300d71b112325d43b99b6148ec7bc5b3 state=OPEN, ts=1381785392864,
server=hor13n08.gq1.ygridcore.net,60020,1381785385596} .. Cannot transit it to
OFFLINE.
java.lang.IllegalStateException: Unexpected state :
{300d71b112325d43b99b6148ec7bc5b3 state=OPEN, ts=1381785392864,
server=hor13n08.gq1.ygridcore.net,60020,1381785385596}
.. Cannot transit it to OFFLINE.
at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:2074)
at
org.apa
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796269#comment-13796269
]
stack commented on HBASE-9776:
--
+1 on trying the patch. We'll soon see if it works or not.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796267#comment-13796267
]
stack commented on HBASE-9776:
--
You mean this?
{code}
Deleting table IntegrationTestLoadAndVerify
2013-10-13 07:42:32,410 INFO [main] hbase.HBaseCluster: Restoring cluster -
started
2013-10-13 07:42:32,414 INFO [main] hbase.HBaseCluster: Added new HBaseAdmin
2013-10-13 07:42:32,414 INFO [main] hbase.HBaseCluster: Restoring cluster -
done
2013-10-13 07:42:32,414 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=35, exceptions:
Sun Oct 13 07:37:26 PDT 2013,
org.apache.hadoop.hbase.client.RpcRetryingCaller@511be529,
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
java.io.IOException: HTableDescriptor missing for IntegrationTestLoadAndVerify
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescriptor(TableEventHandler.java:231)
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.prepareWithTableLock(DeleteTableHandler.java:58)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.prepare(TableEventHandler.java:93)
at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1816)
at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1826)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38213)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
Sun Oct 13 07:37:26 PDT 2013,
org.apache.hadoop.hbase.client.RpcRetryingCaller@511be529,
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
java.io.IOException: HTableDescriptor missing for IntegrationTestLoadAndVerify
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.getTableDescriptor(TableEventHandler.java:231)
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.prepareWithTableLock(DeleteTableHandler.java:58)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.prepare(TableEventHandler.java:93)
at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1816)
at org.apache.hadoop.hbase.master.HMaster.deleteTable(HMaster.java:1826)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:38213)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2146)
at
org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1851)
{code}
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-9776:
-
Attachment: hbase-9776.patch
The fix is simple and use HBaseTestingUtility#deleteTable to delete a table in
clean up phase. The utility deleteTable ignores the TableNotEnabledException
and proceed with deletion.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9775) Client write path scales very badly with more servers
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-9775: - Priority: Critical (was: Major) > Client write path scales very badly with more servers > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Priority: Critical > Attachments: Charts Search Cloudera Manager.png, short_ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9775) Client write path scales very badly with more servers
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-9775: - Attachment: Charts Search Cloudera Manager.png Here's the network. The client barely reaches 1.2 gbit of a possible 10 gbit. We're also seeing a pretty huge slow down vs 0.94. Something in the order of 15% slower on the write path (the write path is better or the same). So I'm thinking that it's got to be something in the AsyncProcess code. Batching seems suspect but I'm running more tests to check. For me that huge a pref loss and a scaling loss are pretty huge issues. > Client write path scales very badly with more servers > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark > Attachments: Charts Search Cloudera Manager.png, short_ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-9776:
-
Status: Patch Available (was: Open)
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
> Attachments: hbase-9776.patch
>
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796254#comment-13796254
]
Hadoop QA commented on HBASE-9773:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608601/trunk-9773_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7557//console
This message is automatically generated.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbas
[jira] [Updated] (HBASE-9606) Apply small scan to meta scan where rowLimit is low
[ https://issues.apache.org/jira/browse/HBASE-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9606: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Apply small scan to meta scan where rowLimit is low > --- > > Key: HBASE-9606 > URL: https://issues.apache.org/jira/browse/HBASE-9606 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.0 > > Attachments: 9606-v2.txt, small-v3.txt > > > HBASE-9488 added the feature for small scan where RPC calls are reduced. > We can apply small scan to MetaScanner#metaScan() where rowLimit is low. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796252#comment-13796252
]
stack commented on HBASE-9773:
--
Patch looks good [~jxiang] Should the state check be tighter than just
checking for null? What could it be? What would be legit and what not?
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9544) Remove TestAdmin#testIsEnabledOnNonexistentTable()
[ https://issues.apache.org/jira/browse/HBASE-9544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9544: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Remove TestAdmin#testIsEnabledOnNonexistentTable() > -- > > Key: HBASE-9544 > URL: https://issues.apache.org/jira/browse/HBASE-9544 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 0.98.0 > > Attachments: 9544.txt > > > TestAdmin#testIsEnabledOnNonexistentTable() is no longer needed > TestAdmin#testIsEnabledOrDisabledOnUnknownTable() covers the above test -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796249#comment-13796249
]
Ted Yu commented on HBASE-9754:
---
In StoreScanner.java, I replaced the initial value with the following and reran
test suite:
{code}
+ private long readPt = Long.MAX_VALUE;
{code}
The suite passed - readPt would be assigned correctly in ctor's so its initial
value doesn't matter.
> Consider eliminating threadlocals from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-rp-0.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9775) Client write path scales very badly with more servers
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796250#comment-13796250 ] stack commented on HBASE-9775: -- What you suspect the issue Mr [~eclark]? Client. The way we are batching up edits in client and sending them across? > Client write path scales very badly with more servers > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark > Attachments: short_ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796251#comment-13796251
]
Ted Yu commented on HBASE-9773:
---
+1
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796245#comment-13796245
]
Hadoop QA commented on HBASE-9773:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608601/trunk-9773_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7556//console
This message is automatically generated.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbas
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796242#comment-13796242 ] Vladimir Rodionov commented on HBASE-9769: -- In 0.94.12 the difference is not so dramatic as in 0.94.6 but still exists: default: 500K rows per sec filter-based: 1.2M rows per sec It seems that there is performance regression in scan filters in 0.94.12. The code which gives me almost 1.5M in 0.94.6 runs only 1.2M rows per sec in 0.94.12. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9771) [WebUI] Humanize store and blockcache statistics on RS
[ https://issues.apache.org/jira/browse/HBASE-9771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9771: Resolution: Fixed Fix Version/s: 0.96.1 0.98.0 Release Note: Committed to 0.96 and trunk. Thanks for the review Elliott. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > [WebUI] Humanize store and blockcache statistics on RS > -- > > Key: HBASE-9771 > URL: https://issues.apache.org/jira/browse/HBASE-9771 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Fix For: 0.98.0, 0.96.1 > > Attachments: HBASE-9771.00.patch, HBASE-9771.01.patch, > HBASE-9771.02.patch > > > Some statistics reported on webui don't include hints about what the unit is, > leaving people guessing about what they mean. Clean them up. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796208#comment-13796208
]
Jeffrey Zhong commented on HBASE-9776:
--
Yes, the disableTable is not idempotent operations because the subsequent retry
fails while we can't remove the enable state check because we use it to sync
operations like: one is trying to do schema changes and the other is trying to
delete the same table.
My plan is to use HBaseTestingUtility#deleteTable instead to let the
application client to eat the exception if happens and proceed with delete
because we're in clean up phase.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-9776:
-
Description:
Occasionally IntegrationTestLoadAndVerify failed with the following error. This
is caused by RPC retry and the first attempt actually went through successfully
and the second retry attempt fails because the table is disabled by the first
attempt.
{code}
2013-10-10
19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: IntegrationTestLoadAndVerify
2013-10-10 19:55:54,340|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
2013-10-10 19:55:54,341|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
2013-10-10 19:55:54,342|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
{code}
was:
Occasionally IntegrationTestLoadAndVerify failed with the following error. This
is caused by RPC retry and the first attempt actually went through
successfully.
{code}
2013-10-10
19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: IntegrationTestLoadAndVerify
2013-10-10 19:55:54,340|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
2013-10-10 19:55:54,341|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
2013-10-10 19:55:54,342|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
{code}
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully and the second retry attempt fails because the table is disabled
> by the first attempt.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-9776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796148#comment-13796148
]
Elliott Clark commented on HBASE-9776:
--
Yep we're seeing this too on our clusters. You can work around this by turning
off deleting the table on the test and using the shell. But the async delete
just seems broken right now.
> Test Load And Verify Fails with TableNotEnabledException
>
>
> Key: HBASE-9776
> URL: https://issues.apache.org/jira/browse/HBASE-9776
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.96.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Minor
>
> Occasionally IntegrationTestLoadAndVerify failed with the following error.
> This is caused by RPC retry and the first attempt actually went through
> successfully.
> {code}
> 2013-10-10
> 19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException:
> IntegrationTestLoadAndVerify
> 2013-10-10 19:55:54,340|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
> 2013-10-10 19:55:54,341|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
> 2013-10-10 19:55:54,342|beaver.machine|INFO|at
> org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (HBASE-9775) Client write path scales very badly with more servers
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795787#comment-13795787 ] Elliott Clark edited comment on HBASE-9775 at 10/15/13 11:20 PM: - Here's what I'm seeing on a largish test cluster. 77 Nodes (the pinkish red) has about the same throughput as the 31 node cluster (blue). They are the exact same machines, I just took away two racks. This is YCSB loads. was (Author: eclark): Here's what I'm seeing on a largish test cluster. 77 Nodes (the pinkish red) has about the same throughput as the 31 node cluster (blue). They are the exact same machines, I just took away two racks. > Client write path scales very badly with more servers > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark > Attachments: short_ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9775) Client write path scales very badly with more servers
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-9775: - Attachment: short_ycsb.png Here's what I'm seeing on a largish test cluster. 77 Nodes (the pinkish red) has about the same throughput as the 31 node cluster (blue). They are the exact same machines, I just took away two racks. > Client write path scales very badly with more servers > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark > Attachments: short_ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9775) Client write path scales very badly with more servers
Elliott Clark created HBASE-9775: Summary: Client write path scales very badly with more servers Key: HBASE-9775 URL: https://issues.apache.org/jira/browse/HBASE-9775 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0 Reporter: Elliott Clark Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9776) Test Load And Verify Fails with TableNotEnabledException
Jeffrey Zhong created HBASE-9776:
Summary: Test Load And Verify Fails with TableNotEnabledException
Key: HBASE-9776
URL: https://issues.apache.org/jira/browse/HBASE-9776
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Priority: Minor
Occasionally IntegrationTestLoadAndVerify failed with the following error. This
is caused by RPC retry and the first attempt actually went through
successfully.
{code}
2013-10-10
19:55:54,339|beaver.machine|INFO|org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: IntegrationTestLoadAndVerify
2013-10-10 19:55:54,340|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:100)
2013-10-10 19:55:54,341|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1979)
2013-10-10 19:55:54,342|beaver.machine|INFO|at
org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1990)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9774) Provide a way for coprocessors to register and report custom metrics
Gary Helmling created HBASE-9774: Summary: Provide a way for coprocessors to register and report custom metrics Key: HBASE-9774 URL: https://issues.apache.org/jira/browse/HBASE-9774 Project: HBase Issue Type: New Feature Components: Coprocessors, metrics Reporter: Gary Helmling It would help provide better visibility into what coprocessors are doing if we provided a way for coprocessors to export their own metrics. The general idea is to: * extend access to the HBase "metrics bus" down into the coprocessor environments * coprocessors can then register and increment custom metrics * coprocessor metrics are then reported along with all others through normal mechanisms -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9771) [WebUI] Humanize store and blockcache statistics on RS
[
https://issues.apache.org/jira/browse/HBASE-9771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795776#comment-13795776
]
Hadoop QA commented on HBASE-9771:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608575/HBASE-9771.02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7554//console
This message is automatically generated.
> [WebUI] Humanize store and blockcache statistics on RS
> --
>
> Key: HBASE-9771
> URL: https://issues.apache.org/jira/browse/HBASE-9771
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Affects Versions: 0.98.0, 0.96.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Trivial
> Attachments: HBASE-9771.00.patch, HBASE-9771.01.patch,
> HBASE-9771.02.patch
>
>
> Some statistics reported on webui don't include hints about what the unit is,
> leaving people guessing about what they mean. Clean them up.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9773:
---
Attachment: trunk-9773_v2.patch
Attached a wrong patch file. Here is the right one.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795774#comment-13795774
]
Hadoop QA commented on HBASE-9773:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608596/trunk-9773.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7555//console
This message is automatically generated.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795754#comment-13795754
]
Jimmy Xiang commented on HBASE-9773:
It's introduced by HBASE-9514 I think. I attached a patch to fix it, added a
test too.
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
> Attachments: trunk-9773.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang reassigned HBASE-9773:
--
Assignee: Jimmy Xiang
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>Assignee: Jimmy Xiang
> Attachments: trunk-9773.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9773:
---
Status: Patch Available (was: Open)
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
> Attachments: trunk-9773.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
[
https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-9773:
---
Attachment: trunk-9773.patch
> Master aborted when hbck asked the master to assign a region that was already
> online
>
>
> Key: HBASE-9773
> URL: https://issues.apache.org/jira/browse/HBASE-9773
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
> Attachments: trunk-9773.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version
> created on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace
> region due to some security exceptions. hbck INCORRECTLY assumed the region
> was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta =>
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a., hdfs =>
> hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
> deployed => } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO [RpcServer.handler=4,port=6]
> master.HMaster: Client=hbase//172.18.145.105 assign
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer
> hosting namespace region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: Sent CLOSE to
> gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
> region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server,
> and in the process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=6]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.; generated
> random
> plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
> src=,
> dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807;
> 4 (online=4, available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Master server abort: loaded coprocessors are:
> [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=6]
> master.HMaster: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
> state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state :
> {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344,
> server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
> .. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK,
> boolean forceNewPlan){code} is the method that does all the above. This was
> called from the HMaster with true for both the boolean arguments.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795737#comment-13795737 ] Jonathan Hsieh commented on HBASE-5487: --- [~sershe] Looking forward to it! > Generic framework for Master-coordinated tasks > -- > > Key: HBASE-5487 > URL: https://issues.apache.org/jira/browse/HBASE-5487 > Project: HBase > Issue Type: New Feature > Components: master, regionserver, Zookeeper >Affects Versions: 0.94.0 >Reporter: Mubarak Seyed >Priority: Critical > Attachments: Region management in Master.pdf > > > Need a framework to execute master-coordinated tasks in a fault-tolerant > manner. > Master-coordinated tasks such as online-scheme change and delete-range > (deleting region(s) based on start/end key) can make use of this framework. > The advantages of framework are > 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for > master-coordinated tasks > 2. Ability to abstract the common functions across Master -> ZK and RS -> ZK > 3. Easy to plugin new master-coordinated tasks without adding code to core > components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9753) Excessive readpoint checks in MemstoreScanner
[ https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795732#comment-13795732 ] Hudson commented on HBASE-9753: --- SUCCESS: Integrated in hbase-0.96 #141 (See [https://builds.apache.org/job/hbase-0.96/141/]) HBASE-9753 Excessive readpoint checks in MemstoreScanner (tedyu: rev 1532431) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java > Excessive readpoint checks in MemstoreScanner > - > > Key: HBASE-9753 > URL: https://issues.apache.org/jira/browse/HBASE-9753 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Ted Yu > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9753-0.94.txt, 9753-v1.txt > > > Brought up by [~vrodionov] on the mailing list. See also HBASE-9751. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[
https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795707#comment-13795707
]
Hadoop QA commented on HBASE-9748:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608354/trunk-9748.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7553//console
This message is automatically generated.
> Address outstanding comments raised for HBASE-9696
> --
>
> Key: HBASE-9748
> URL: https://issues.apache.org/jira/browse/HBASE-9748
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9748.patch
>
>
> This is a follow-up issue of HBASE-9696.
> There are some outstanding review comments in
> https://reviews.apache.org/r/14470/ that haven't been addressed. I will
> address them later in this jira.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795694#comment-13795694 ] Lars Hofhansl commented on HBASE-9769: -- Also try with 0.94.12. The specific issue you're seeing might be fixed there. > Improve performance of a Scanner with explicit column list when rows are > small/medium size > -- > > Key: HBASE-9769 > URL: https://issues.apache.org/jira/browse/HBASE-9769 > Project: HBase > Issue Type: Improvement > Components: Scanners >Affects Versions: 0.98.0, 0.94.12, 0.96.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9771) [WebUI] Humanize store and blockcache statistics on RS
[ https://issues.apache.org/jira/browse/HBASE-9771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795683#comment-13795683 ] Elliott Clark commented on HBASE-9771: -- +1 lgtm > [WebUI] Humanize store and blockcache statistics on RS > -- > > Key: HBASE-9771 > URL: https://issues.apache.org/jira/browse/HBASE-9771 > Project: HBase > Issue Type: Improvement > Components: UI >Affects Versions: 0.98.0, 0.96.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Attachments: HBASE-9771.00.patch, HBASE-9771.01.patch, > HBASE-9771.02.patch > > > Some statistics reported on webui don't include hints about what the unit is, > leaving people guessing about what they mean. Clean them up. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795680#comment-13795680 ] Sergey Shelukhin commented on HBASE-5487: - [~jmhsieh] I am writing out very detailed operation and failover descriptions right now :) > Generic framework for Master-coordinated tasks > -- > > Key: HBASE-5487 > URL: https://issues.apache.org/jira/browse/HBASE-5487 > Project: HBase > Issue Type: New Feature > Components: master, regionserver, Zookeeper >Affects Versions: 0.94.0 >Reporter: Mubarak Seyed >Priority: Critical > Attachments: Region management in Master.pdf > > > Need a framework to execute master-coordinated tasks in a fault-tolerant > manner. > Master-coordinated tasks such as online-scheme change and delete-range > (deleting region(s) based on start/end key) can make use of this framework. > The advantages of framework are > 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for > master-coordinated tasks > 2. Ability to abstract the common functions across Master -> ZK and RS -> ZK > 3. Easy to plugin new master-coordinated tasks without adding code to core > components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9772) Normalize new client default values
[
https://issues.apache.org/jira/browse/HBASE-9772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13795679#comment-13795679
]
Hadoop QA commented on HBASE-9772:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608553/HBASE-9772.00.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7552//console
This message is automatically generated.
> Normalize new client default values
> ---
>
> Key: HBASE-9772
> URL: https://issues.apache.org/jira/browse/HBASE-9772
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 0.98.0, 0.96.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Attachments: HBASE-9772.00.patch
>
>
> The new AsyncProcess exposes a handful of configuration settings for
> controlling the number and distribution of concurrent connections across
> regionservers. Use default values consistently and bubble up their usage to
> users.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
