[jira] [Resolved] (HBASE-12713) IncreasingToUpperBoundRegionSplitPolicy invalid when region count greater than 100
[ https://issues.apache.org/jira/browse/HBASE-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Junhong resolved HBASE-12713. - Resolution: Duplicate Simple as HBASE-12451 IncreasingToUpperBoundRegionSplitPolicy invalid when region count greater than 100 -- Key: HBASE-12713 URL: https://issues.apache.org/jira/browse/HBASE-12713 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.98.6 Reporter: Liu Junhong Original Estimate: 4m Remaining Estimate: 4m I find that the policy of region split which in IncreasingToUpperBoundRegionSplitPolicy will be invalid when the count of region is greater than 100. But sometimes 100 regions is not too much for a cluster that has 50 or more regionservers. So i think this policy should consider the density of the regions but not the total count of the regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12710) TestAssignmentManager may fail due to concurrent tests
[
https://issues.apache.org/jira/browse/HBASE-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251373#comment-14251373
]
Dima Spivak commented on HBASE-12710:
-
Correct me if I'm wrong, [~tedyu], but aren't MediumTests and LargeTests
categories run together as part of -PrunAllTests? [According to the
pom|https://github.com/apache/hbase/blob/master/pom.xml#L2073], these
categories both have a default fork count of 5, so I'm not sure this change
would actually correct the problem. I think we should instead look at why
[SmallTests are run with no
forking|https://github.com/apache/hbase/blob/master/pom.xml#L2068], while
Medium and Large do and, as you observed, experience spurious failures because
of this concurrency.
TestAssignmentManager may fail due to concurrent tests
--
Key: HBASE-12710
URL: https://issues.apache.org/jira/browse/HBASE-12710
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments: 12710-1.0.txt,
org.apache.hadoop.hbase.master.TestAssignmentManager-output.txt
I saw TestAssignmentManager#testMasterRestartWhenTableInEnabling fail when
running test suite on branch-1:
{code}
testMasterRestartWhenTableInEnabling(org.apache.hadoop.hbase.master.TestAssignmentManager)
Time elapsed: 0.714 sec FAILURE!
java.lang.AssertionError: Table should be enabled.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.master.TestAssignmentManager.testMasterRestartWhenTableInEnabling(TestAssignmentManager.java:1023)
{code}
In test output, I saw:
{code}
2014-12-18 00:45:18,230 FATAL [kiyo:38816.activeMasterManager]
master.HMaster(1802): Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: Illegal character 10 at 0. Namespaces
can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9]:
^Ehbase^R^Dmeta
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:215)
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:204)
at org.apache.hadoop.hbase.TableName.init(TableName.java:307)
at
org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:344)
at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:465)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2686)
at
org.apache.hadoop.hbase.HTableDescriptor.convert(HTableDescriptor.java:1525)
at
org.apache.hadoop.hbase.HTableDescriptor.parseFrom(HTableDescriptor.java:1487)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:526)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:730)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:706)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:693)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:466)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:145)
at
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:125)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:592)
{code}
But running TestAssignmentManager standalone, I don't see the above exception.
Since TestAssignmentManager is a medium test, the above exception was likely
caused by concurrent tests.
TestAssignmentManager should be categorized as large test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'
[ https://issues.apache.org/jira/browse/HBASE-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaiyu Zhu updated HBASE-12511: --- Attachment: HBASE-12511.patch namespace permissions - add support from table creation privilege in a namespace 'C' Key: HBASE-12511 URL: https://issues.apache.org/jira/browse/HBASE-12511 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Huaiyu Zhu Fix For: 1.1.0 Attachments: HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch As discussed in namespace permission Jira. A user granted a 'C' on a namespace enables a user to create tables within the namespace. 'C' on a namespace does not enable a user to create/drop the namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'
[
https://issues.apache.org/jira/browse/HBASE-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251386#comment-14251386
]
Hadoop QA commented on HBASE-12511:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687989/HBASE-12511.patch
against master branch at commit 83e4bfaf73e1c7db16835b20c4f996adde30054a.
ATTACHMENT ID: 12687989
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12133//console
This message is automatically generated.
namespace permissions - add support from table creation privilege in a
namespace 'C'
Key: HBASE-12511
URL: https://issues.apache.org/jira/browse/HBASE-12511
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Huaiyu Zhu
Fix For: 1.1.0
Attachments: HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch,
HBASE-12511.patch
As discussed in namespace permission Jira. A user granted a 'C' on a
namespace enables a user to create tables within the namespace. 'C' on a
namespace does not enable a user to create/drop the namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'
[ https://issues.apache.org/jira/browse/HBASE-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251388#comment-14251388 ] Huaiyu Zhu commented on HBASE-12511: As we agreed: 1. table creation requires minimal NS 'C'. 2. other table managements (delete/modify/etc) require minimal table 'A|C' 3. namespace managements require minimal global 'A' 4. 'L' is not implemented here I fixed the requireGlobalPermission so that it only checks global. Added a new method requireNamespacePermission for checking namespace. modify the unit tests to verify: 1. if you have NS 'C', you can create table under the namespace 2. if you have only NS 'A', you CANNOT create table under the namespace 3. if you have only NS 'A', you CANNOT grant/revoke namespace permissions or modify namespace namespace permissions - add support from table creation privilege in a namespace 'C' Key: HBASE-12511 URL: https://issues.apache.org/jira/browse/HBASE-12511 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Huaiyu Zhu Fix For: 1.1.0 Attachments: HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch As discussed in namespace permission Jira. A user granted a 'C' on a namespace enables a user to create tables within the namespace. 'C' on a namespace does not enable a user to create/drop the namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12028) Abort the RegionServer, when one of it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alicia Ying Shu updated HBASE-12028: Attachment: (was: Phoenix-1409-v1.patch) Abort the RegionServer, when one of it's handler threads die Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12028) Abort the RegionServer, when one of it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alicia Ying Shu updated HBASE-12028: Attachment: Phoenix-1409-v1.patch Abort the RegionServer, when one of it's handler threads die Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'
[ https://issues.apache.org/jira/browse/HBASE-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251402#comment-14251402 ] Srikanth Srungarapu commented on HBASE-12511: - Can a global admin can create table? namespace permissions - add support from table creation privilege in a namespace 'C' Key: HBASE-12511 URL: https://issues.apache.org/jira/browse/HBASE-12511 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Huaiyu Zhu Fix For: 1.1.0 Attachments: HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch As discussed in namespace permission Jira. A user granted a 'C' on a namespace enables a user to create tables within the namespace. 'C' on a namespace does not enable a user to create/drop the namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12511) namespace permissions - add support from table creation privilege in a namespace 'C'
[
https://issues.apache.org/jira/browse/HBASE-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251412#comment-14251412
]
Huaiyu Zhu commented on HBASE-12511:
{quote}Can a global admin can create table?{quote}
No. but global admin can delete/modify the table.
namespace permissions - add support from table creation privilege in a
namespace 'C'
Key: HBASE-12511
URL: https://issues.apache.org/jira/browse/HBASE-12511
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Huaiyu Zhu
Fix For: 1.1.0
Attachments: HBASE-12511.patch, HBASE-12511.patch, HBASE-12511.patch,
HBASE-12511.patch
As discussed in namespace permission Jira. A user granted a 'C' on a
namespace enables a user to create tables within the namespace. 'C' on a
namespace does not enable a user to create/drop the namespace.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12673) Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir
[ https://issues.apache.org/jira/browse/HBASE-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251411#comment-14251411 ] Jiajia Li commented on HBASE-12673: --- Hi, [~j...@cloudera.com], can you look at above comment and give some advise? Thanks Add a UT to read mob file when the mob hfile moving from the mob dir to the archive dir --- Key: HBASE-12673 URL: https://issues.apache.org/jira/browse/HBASE-12673 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Affects Versions: hbase-11339 Reporter: Jiajia Li Assignee: Jiajia Li Fix For: hbase-11339 Attachments: HBASE-12673.patch add a unit test to scan the cloned table when deleting the original table, and the steps as following: 1) create a table with mobs, 2) snapshot it, 3) clone it as a a different table 4) have a read workload on the snapshot 5) delete the original table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11412) Minimize a number of hbase-client transitive dependencies
[
https://issues.apache.org/jira/browse/HBASE-11412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251435#comment-14251435
]
Sergey Beryozkin commented on HBASE-11412:
--
Hi Enis, thanks for applying the patch; I see a couple of tests have failed - I
can't link those failures to this patch though - let me know please if you
reckon the patch needs more work. As a side note, more dependencies can likely
be excluded but I've no enough HBase experience at this stage to optimize it
further; will try to return to this issue when we start getting some feedback
from our users
Cheers, Sergey
Minimize a number of hbase-client transitive dependencies
-
Key: HBASE-11412
URL: https://issues.apache.org/jira/browse/HBASE-11412
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.98.3
Reporter: Sergey Beryozkin
Assignee: Sergey Beryozkin
Priority: Minor
Fix For: 1.0.0, 2.0.0
Attachments: hbase11412.txt, hbase11412_2.patch, hbase11412_3.patch
hbase-client has a number of transitive dependencies not needed for a client
mode execution. In my test I've added the following exclusions:
{code:xml}
exclusions
exclusion
groupIdcom.sun.jersey/groupId
artifactIdjersey-server/artifactId
/exclusion
exclusion
groupIdcom.sun.jersey/groupId
artifactIdjersey-core/artifactId
/exclusion
exclusion
groupIdcom.sun.jersey/groupId
artifactIdjersey-json/artifactId
/exclusion
exclusion
groupIdcom.sun.jersey.contribs/groupId
artifactIdjersey-guice/artifactId
/exclusion
exclusion
groupIdcom.google.inject/groupId
artifactIdguice/artifactId
/exclusion
exclusion
groupIdcom.google.inject.extensions/groupId
artifactIdguice-servlet/artifactId
/exclusion
exclusion
groupIdorg.mortbay.jetty/groupId
artifactIdjetty/artifactId
/exclusion
exclusion
groupIdorg.mortbay.jetty/groupId
artifactIdjetty-util/artifactId
/exclusion
exclusion
groupIdcommons-httpclient/groupId
artifactIdcommons-httpclient/artifactId
/exclusion
/exclusions
{code}
Proposal: add related exclusions to some of the dependencies hbase-client
depends upon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuanBo Peng updated HBASE-12223: Attachment: HBASE-12223-0.94.patch MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[
https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251475#comment-14251475
]
Hadoop QA commented on HBASE-12223:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688008/HBASE-12223-0.94.patch
against master branch at commit 83e4bfaf73e1c7db16835b20c4f996adde30054a.
ATTACHMENT ID: 12688008
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12134//console
This message is automatically generated.
MultiTableInputFormatBase.getSplits is too slow
---
Key: HBASE-12223
URL: https://issues.apache.org/jira/browse/HBASE-12223
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.94.15
Reporter: shanwen
Assignee: YuanBo Peng
Priority: Minor
Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0, 0.94.27
Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch,
HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch,
HBASE-12223-v4.patch, HBASE-12223.patch
when use Multiple scan,getSplits is too slow,800 scans take five minutes
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12715) Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName
zhangduo created HBASE-12715: Summary: Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName Key: HBASE-12715 URL: https://issues.apache.org/jira/browse/HBASE-12715 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.98.9 Reporter: zhangduo When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it is renamed in master branch, doesn't matter) already returns -1, then I found that we use encodedRegionName when calling this method. But when we doing regionServerReport, we use regionName to report to HMaster(see the code in HRegionServer.createRegionLoad). We need to fix this, as seems we will disable distributed log replay in 0.98 by default? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12715) Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName
[ https://issues.apache.org/jira/browse/HBASE-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangduo updated HBASE-12715: - Description: When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it is renamed in master branch, doesn't matter) always returns -1, then I found that we use encodedRegionName when calling this method. But when we doing regionServerReport, we use regionName to report to HMaster(see the code in HRegionServer.createRegionLoad). We need to fix this, as seems we will disable distributed log replay in 0.98 by default? was: When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it is renamed in master branch, doesn't matter) already returns -1, then I found that we use encodedRegionName when calling this method. But when we doing regionServerReport, we use regionName to report to HMaster(see the code in HRegionServer.createRegionLoad). We need to fix this, as seems we will disable distributed log replay in 0.98 by default? Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName -- Key: HBASE-12715 URL: https://issues.apache.org/jira/browse/HBASE-12715 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.98.9 Reporter: zhangduo When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it is renamed in master branch, doesn't matter) always returns -1, then I found that we use encodedRegionName when calling this method. But when we doing regionServerReport, we use regionName to report to HMaster(see the code in HRegionServer.createRegionLoad). We need to fix this, as seems we will disable distributed log replay in 0.98 by default? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: HBASE-12684-v4.patch - Changed AsyncRpcChannel to implement BlockingRpcChannel - AsyncRpcClient now returns an AsyncRpcChannel with method createBlockingRpcChannel. Saves BlockingRpcChannelImplementation creation on each request. - To enable this I had to include the ServiceName in the createBlockingRpcChannel method. Refactored RpcClientImpl and all calls in ConnectionManager and tests to work correctly. - Enlarged the max frame size to Integer.MAX in LengthFieldBasedFrameDecoder of the AsyncRpcClient since I encountered the previous max in one test run. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-v1.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12715) Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName
[ https://issues.apache.org/jira/browse/HBASE-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251579#comment-14251579 ] zhangduo commented on HBASE-12715: -- Seems that HLogKey does not have a getRegionName method... So should we use encodedRegionName when doing regionServerReport? And it is hard to write a testcase because we skip edits everywhere. It does not impact correctness, only impacts MTTR. Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName -- Key: HBASE-12715 URL: https://issues.apache.org/jira/browse/HBASE-12715 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.98.9 Reporter: zhangduo When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it is renamed in master branch, doesn't matter) always returns -1, then I found that we use encodedRegionName when calling this method. But when we doing regionServerReport, we use regionName to report to HMaster(see the code in HRegionServer.createRegionLoad). We need to fix this, as seems we will disable distributed log replay in 0.98 by default? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12223: --- Fix Version/s: (was: 1.1.0) (was: 1.0.1) 1.0.0 Hadoop Flags: Reviewed Looks like 1.0 RC hasn't been cut. I am integrating to 4 branches. MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251630#comment-14251630
]
Hadoop QA commented on HBASE-12684:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688009/HBASE-12684-v4.patch
against master branch at commit 83e4bfaf73e1c7db16835b20c4f996adde30054a.
ATTACHMENT ID: 12688009
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2098 checkstyle errors (more than the master's current 2087 errors).
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestReplicasClient
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation
org.apache.hadoop.hbase.client.TestReplicaWithCluster
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRSCompressed
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.replication.TestReplicationKillMasterRS
org.apache.hadoop.hbase.client.TestFastFail
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.replication.TestReplicationKillSlaveRS
{color:red}-1 core zombie tests{color}. There are 8 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:246)
at
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush.testLogReplay(TestPerColumnFamilyFlush.java:413)
at
org.apache.hadoop.hbase.client.TestFromClientSide.testMiscHTableStuff(TestFromClientSide.java:4194)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:228)
at
org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:258)
at
org.apache.hadoop.hbase.TestZooKeeper.testMasterZKSessionRecoveryFailure(TestZooKeeper.java:243)
at
org.apache.hadoop.hbase.client.TestHCM.testConnectionClose(TestHCM.java:340)
at
org.apache.hadoop.hbase.client.TestHCM.testConnectionCloseAllowsInterrupt(TestHCM.java:293)
at
org.apache.hadoop.hbase.client.TestFromClientSide.testMajorCompactionBetweenTwoUpdates(TestFromClientSide.java:3699)
at
org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.testCFKeyRotation(TestEncryptionKeyRotation.java:103)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12135//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251639#comment-14251639 ] YuanBo Peng commented on HBASE-12223: - thank you very much,ted yu MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
Weichen Ye created HBASE-12716:
--
Summary: A bug in RegionSplitter.UniformSplit algorithm
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Weichen Ye
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
for (byte b : mid) {
System.out.print(b + );
}
System.out.println(\n + new String(mid));
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
at test.ByteMid.main(ByteMid.java:196)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Ye updated HBASE-12716:
---
Description:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
for (byte b : mid) {
System.out.print(b + );
}
System.out.println(\n + new String(mid));
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
at test.ByteMid.main(ByteMid.java:196)
was:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
for (byte b : mid) {
System.out.print(b + );
}
System.out.println(\n + new String(mid));
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
at test.ByteMid.main(ByteMid.java:196)
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Weichen Ye
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
for (byte b : mid) {
System.out.print(b + );
}
System.out.println(\n + new String(mid));
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
at test.ByteMid.main(ByteMid.java:196)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12717) Pre-split in HBaseAdmin.create() can not find the split point
Weichen Ye created HBASE-12717: -- Summary: Pre-split in HBaseAdmin.create() can not find the split point Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) the pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. This pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12717) Pre-split in HBaseAdmin.create() can not find the split point
[ https://issues.apache.org/jira/browse/HBASE-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12717: --- Description: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. This pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) was: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) the pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. This pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) Pre-split in HBaseAdmin.create() can not find the split point - Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. This pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12717) Pre-split in HBaseAdmin.create() can not find the split point
[ https://issues.apache.org/jira/browse/HBASE-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12717: --- Description: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P was: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. This pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) Pre-split in HBaseAdmin.create() can not find the split point - Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12717) Pre-split in HBaseAdmin.create() can not find the split point
[ https://issues.apache.org/jira/browse/HBASE-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12717: --- Description: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to get split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P was: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to get split point with an additional bytes. for example: aaa and aab, split point= aaaP and 1112, split point =P Pre-split in HBaseAdmin.create() can not find the split point - Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to get split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job
[ https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251686#comment-14251686 ] Weichen Ye commented on HBASE-12590: [~j...@cloudera.com] Hi~ I used to try the algorithm in RegionSplitter, but I find there is a small bug. If the start key is the same length as the end key, and their last bytes are adjacent in alphabetical order , the algorithm would not calculate a split point with an additional byte. This split algorithm is not very related to the data skew in HBase-MapReduce job, so i create two new issues about it . https://issues.apache.org/jira/browse/HBASE-12716 https://issues.apache.org/jira/browse/HBASE-12717 A solution for data skew in HBase-Mapreduce Job --- Key: HBASE-12590 URL: https://issues.apache.org/jira/browse/HBASE-12590 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Weichen Ye Attachments: A Solution for Data Skew in HBase-MapReduce Job (Version2).pdf, A Solution for Data Skew in HBase-MapReduce Job (Version3).pdf, HBASE-12590-v3.patch, HBASE-12590-v4.patch, HBase-12590-v1.patch, HBase-12590-v2.patch 1, Motivation In production environment, data skew is a very common case. A HBase table may contains a lot of small regions and several large regions. Small regions waste a lot of computing resources. If we use a job to scan a table with 3000 small regions, we need a job with 3000 mappers. Large regions always block the job. If in a 100-region table, one region is far large then the other 99 regions. When we run a job with the table as input, 99 mappers will be completed very quickly, and then we need to wait for the last mapper for a long time. 2, Configuration Add three new configuration hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in HBase-MapReduce jobs. The default value is false. hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region size is larger than 3x average region size, treat the region as “proportionately too large”. hbase.table.row.textkey = true means the row key is text. False means binary row key. It is used to find the mid row key in large region. The default value is true. If (region size = average size*ratio) : cut the region into two MR input splits If (average size = region size average size*ratio) : one region as one MR input split If (sum of several continuous regions size average size): combine these regions into one MR input split. Example: In attachment Welcome to the Review Board. https://reviews.apache.org/r/28494/diff/# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12717) Pre-split in HBaseAdmin.create() can not find the split point
[ https://issues.apache.org/jira/browse/HBASE-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12717: --- Description: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to calculate the split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P was: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to get split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P Pre-split in HBaseAdmin.create() can not find the split point - Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to calculate the split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12713) IncreasingToUpperBoundRegionSplitPolicy invalid when region count greater than 100
[ https://issues.apache.org/jira/browse/HBASE-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Junhong reopened HBASE-12713: - Sorry for wrong description. IncreasingToUpperBoundRegionSplitPolicy invalid when region count greater than 100 -- Key: HBASE-12713 URL: https://issues.apache.org/jira/browse/HBASE-12713 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.98.6 Reporter: Liu Junhong Original Estimate: 4m Remaining Estimate: 4m I find that the policy of region split which in IncreasingToUpperBoundRegionSplitPolicy will be invalid when the count of region is greater than 100. But sometimes 100 regions is not too much for a cluster that has 50 or more regionservers. So i think this policy should consider the density of the regions but not the total count of the regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Ye updated HBASE-12716:
---
Description:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
--
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
was:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
###33
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Weichen Ye
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
--
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weichen Ye updated HBASE-12716:
---
Description:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
###33
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
was:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
for (byte b : mid) {
System.out.print(b + );
}
System.out.println(\n + new String(mid));
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
at test.ByteMid.main(ByteMid.java:196)
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Weichen Ye
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
###33
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12717) Pre-split in HBaseAdmin.create() can not find the split point
[ https://issues.apache.org/jira/browse/HBASE-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12717: --- Description: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to calculate the split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P was: When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to calculate the split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P Pre-split in HBaseAdmin.create() can not find the split point - Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to calculate the split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12713) IncreasingToUpperBoundRegionSplitPolicy invalid when region count greater than 100
[ https://issues.apache.org/jira/browse/HBASE-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Junhong updated HBASE-12713: Description: I find that the policy of region split which in IncreasingToUpperBoundRegionSplitPolicy will be use the value of maxfilesize when the count of region is greater than 100. But sometimes 100 regions is not too much for a cluster that has 50 or more regionservers. So i think this policy should consider the density of the regions but not the total count of the regions. was: I find that the policy of region split which in IncreasingToUpperBoundRegionSplitPolicy will be invalid when the count of region is greater than 100. But sometimes 100 regions is not too much for a cluster that has 50 or more regionservers. So i think this policy should consider the density of the regions but not the total count of the regions. IncreasingToUpperBoundRegionSplitPolicy invalid when region count greater than 100 -- Key: HBASE-12713 URL: https://issues.apache.org/jira/browse/HBASE-12713 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.98.6 Reporter: Liu Junhong Original Estimate: 4m Remaining Estimate: 4m I find that the policy of region split which in IncreasingToUpperBoundRegionSplitPolicy will be use the value of maxfilesize when the count of region is greater than 100. But sometimes 100 regions is not too much for a cluster that has 50 or more regionservers. So i think this policy should consider the density of the regions but not the total count of the regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12717) Pre-split algorithm in HBaseAdmin.create() can not find the split point
[ https://issues.apache.org/jira/browse/HBASE-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Ye updated HBASE-12717: --- Summary: Pre-split algorithm in HBaseAdmin.create() can not find the split point (was: Pre-split in HBaseAdmin.create() can not find the split point) Pre-split algorithm in HBaseAdmin.create() can not find the split point --- Key: HBASE-12717 URL: https://issues.apache.org/jira/browse/HBASE-12717 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.98.6 Reporter: Weichen Ye Priority: Critical When we set the start key and the end key in the function: createTable(HTableDescriptor desc, byte[] startKey, byte[] endKey, int numRegions) The current pre-split algorithm could not find a split point between the keys like aaa and aab, and 1112. Example Code for this bug: admin.createTable(htd, Bytes.toBytes(aaa), Bytes.toBytes(aab), 4); we will get the following ERROR: Exception in thread main java.lang.IllegalArgumentException: Unable to split key range into enough regions at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:473) at test.JavaTest.main(JavaTest.java:28) We hope this pre-split algorithm should be able to calculate the split point with an additional byte. for example: aaa and aab, split point= aaaP and 1112, split point =P -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251733#comment-14251733 ] Hudson commented on HBASE-12223: FAILURE: Integrated in HBase-1.1 #2 (See [https://builds.apache.org/job/HBase-1.1/2/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev effbe858889d5a89b18e494edd6cf66fd75d2608) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251757#comment-14251757 ] Hudson commented on HBASE-12223: SUCCESS: Integrated in HBase-0.94-security #560 (See [https://builds.apache.org/job/HBase-0.94-security/560/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 20c83b53cac6d45966f8a2e8b7f596a5b4639adb) * src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251777#comment-14251777 ] Hudson commented on HBASE-12223: FAILURE: Integrated in HBase-TRUNK #5943 (See [https://builds.apache.org/job/HBase-TRUNK/5943/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 15cf0a6e7b10882f7fd205b65c2ef265a690597d) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251785#comment-14251785 ] Hudson commented on HBASE-12223: SUCCESS: Integrated in HBase-0.94-JDK7 #215 (See [https://builds.apache.org/job/HBase-0.94-JDK7/215/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 20c83b53cac6d45966f8a2e8b7f596a5b4639adb) * src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251788#comment-14251788 ] Hudson commented on HBASE-12223: FAILURE: Integrated in HBase-0.94 #1447 (See [https://builds.apache.org/job/HBase-0.94/1447/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 20c83b53cac6d45966f8a2e8b7f596a5b4639adb) * src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251791#comment-14251791 ] Hudson commented on HBASE-12223: SUCCESS: Integrated in HBase-0.98 #749 (See [https://builds.apache.org/job/HBase-0.98/749/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 74d9cc8a0c117a3b982d34523fca17b86ff5d0e4) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12718) Convert TestAcidGurantees from a unit test to an integration test
Jonathan Hsieh created HBASE-12718: -- Summary: Convert TestAcidGurantees from a unit test to an integration test Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251825#comment-14251825 ] Hudson commented on HBASE-12223: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #715 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/715/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 74d9cc8a0c117a3b982d34523fca17b86ff5d0e4) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12718) Convert TestAcidGurantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12718: --- Attachment: hbase-12718.patch Convert TestAcidGurantees from a unit test to an integration test - Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
Sean Busbey created HBASE-12719: --- Summary: Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 2.0.0, 1.1.0 Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12718) Convert TestAcidGurantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12718: --- Status: Patch Available (was: Open) Convert TestAcidGurantees from a unit test to an integration test - Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-12719 started by Sean Busbey. --- Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 2.0.0, 1.1.0 Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12718) Convert TestAcidGuarantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12718: --- Summary: Convert TestAcidGuarantees from a unit test to an integration test (was: Convert TestAcidGurantees from a unit test to an integration test) Convert TestAcidGuarantees from a unit test to an integration test -- Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12718) Convert TestAcidGurantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251827#comment-14251827 ] Jonathan Hsieh commented on HBASE-12718: tested using : mvn install -DskipTests cd hbase-it mvn verify -Dit.test=IntegrationTestAcidGuarantees and started a standalone hbase instance and ran bin/hbase org.apache.hadoop.hbase.IntegrationTestAcidGuarantees Convert TestAcidGurantees from a unit test to an integration test - Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12223: --- Fix Version/s: 1.1.0 MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12716:
---
Description:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
was:
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm meet
an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
--
We will get the ERROR:
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Weichen Ye
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251899#comment-14251899
]
Ted Yu commented on HBASE-12716:
[~yeweichen]:
Can you put the code snippet above in TestRegionSplitter#unitTestUniformSplit ?
If you have a patch for the fix, that would be even better.
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.6
Reporter: Weichen Ye
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12710) TestAssignmentManager may fail due to concurrent tests
[
https://issues.apache.org/jira/browse/HBASE-12710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251902#comment-14251902
]
Ted Yu commented on HBASE-12710:
Dima:
The patch didn't solve the problem.
More investigation is needed.
TestAssignmentManager may fail due to concurrent tests
--
Key: HBASE-12710
URL: https://issues.apache.org/jira/browse/HBASE-12710
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Attachments: 12710-1.0.txt,
org.apache.hadoop.hbase.master.TestAssignmentManager-output.txt
I saw TestAssignmentManager#testMasterRestartWhenTableInEnabling fail when
running test suite on branch-1:
{code}
testMasterRestartWhenTableInEnabling(org.apache.hadoop.hbase.master.TestAssignmentManager)
Time elapsed: 0.714 sec FAILURE!
java.lang.AssertionError: Table should be enabled.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.master.TestAssignmentManager.testMasterRestartWhenTableInEnabling(TestAssignmentManager.java:1023)
{code}
In test output, I saw:
{code}
2014-12-18 00:45:18,230 FATAL [kiyo:38816.activeMasterManager]
master.HMaster(1802): Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: Illegal character 10 at 0. Namespaces
can only contain 'alphanumeric characters': i.e. [a-zA-Z_0-9]:
^Ehbase^R^Dmeta
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:215)
at
org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:204)
at org.apache.hadoop.hbase.TableName.init(TableName.java:307)
at
org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:344)
at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:465)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2686)
at
org.apache.hadoop.hbase.HTableDescriptor.convert(HTableDescriptor.java:1525)
at
org.apache.hadoop.hbase.HTableDescriptor.parseFrom(HTableDescriptor.java:1487)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:526)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:730)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:706)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptor(FSTableDescriptors.java:693)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:466)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:145)
at
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:125)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:592)
{code}
But running TestAssignmentManager standalone, I don't see the above exception.
Since TestAssignmentManager is a medium test, the above exception was likely
caused by concurrent tests.
TestAssignmentManager should be categorized as large test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[
https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251919#comment-14251919
]
Ted Yu commented on HBASE-12712:
In master branch, we have:
{code}
if (!(forceMajor isAllFiles)) {
candidateSelection = skipLargeFiles(candidateSelection);
isAllFiles = candidateFiles.size() == candidateSelection.size();
}
{code}
In 0.98, we have:
{code}
if (!forceMajor) {
candidateSelection = skipLargeFiles(candidateSelection);
}
{code}
Can you try 0.98.8 to see if the problem is still there ?
skipLargeFiles in minor compact but not in major compact
Key: HBASE-12712
URL: https://issues.apache.org/jira/browse/HBASE-12712
Project: HBase
Issue Type: New Feature
Components: Compaction
Affects Versions: 0.98.6
Reporter: Liu Junhong
Labels: newbie, patch
Fix For: 0.98.6
Attachments: compact.diff
Original Estimate: 72h
Remaining Estimate: 72h
Here is my case. After repeatedly minor compaction, the size of storefile is
very large. Compaction with large storefile will waste much bandwidth, so i
use the “hbase.hstore.compaction.max.size” to skip this case. But after use
this config, i find that major compaction will be skipped forever when i read
the source code and the deletes and muti-versions data my waste storage. So i
had to modify the code.
Now i'm try to submit my patch.But my patch is not perfect. I think there
should be an other config to determine if the large size storefile should
join major compaction in HColumnDescriptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12684: -- Attachment: HBASE-12684-v5.patch - Fixed issue with client not reacting on connection fail issues. (Should fix some tests) - Fixed issue with client reusing canceled promise on blocking calls. Not all tests are fixed. But lets see what HadoopQA thinks of the progress. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-v1.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[
https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251919#comment-14251919
]
Ted Yu edited comment on HBASE-12712 at 12/18/14 5:30 PM:
--
In master branch, we have:
{code}
if (!(forceMajor isAllFiles)) {
candidateSelection = skipLargeFiles(candidateSelection);
isAllFiles = candidateFiles.size() == candidateSelection.size();
}
{code}
In 0.98, we have:
{code}
if (!forceMajor) {
candidateSelection = skipLargeFiles(candidateSelection);
}
{code}
So what you observed is specific to 0.98 ?
was (Author: yuzhih...@gmail.com):
In master branch, we have:
{code}
if (!(forceMajor isAllFiles)) {
candidateSelection = skipLargeFiles(candidateSelection);
isAllFiles = candidateFiles.size() == candidateSelection.size();
}
{code}
In 0.98, we have:
{code}
if (!forceMajor) {
candidateSelection = skipLargeFiles(candidateSelection);
}
{code}
Can you try 0.98.8 to see if the problem is still there ?
skipLargeFiles in minor compact but not in major compact
Key: HBASE-12712
URL: https://issues.apache.org/jira/browse/HBASE-12712
Project: HBase
Issue Type: New Feature
Components: Compaction
Affects Versions: 0.98.6
Reporter: Liu Junhong
Labels: newbie, patch
Fix For: 0.98.6
Attachments: compact.diff
Original Estimate: 72h
Remaining Estimate: 72h
Here is my case. After repeatedly minor compaction, the size of storefile is
very large. Compaction with large storefile will waste much bandwidth, so i
use the “hbase.hstore.compaction.max.size” to skip this case. But after use
this config, i find that major compaction will be skipped forever when i read
the source code and the deletes and muti-versions data my waste storage. So i
had to modify the code.
Now i'm try to submit my patch.But my patch is not perfect. I think there
should be an other config to determine if the large size storefile should
join major compaction in HColumnDescriptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12712) skipLargeFiles in minor compact but not in major compact
[ https://issues.apache.org/jira/browse/HBASE-12712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12712: --- Release Note: (was: It's my first time to submit a patch in jira. Sorry for mistake.) skipLargeFiles in minor compact but not in major compact Key: HBASE-12712 URL: https://issues.apache.org/jira/browse/HBASE-12712 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.98.6 Reporter: Liu Junhong Labels: newbie, patch Fix For: 0.98.6 Attachments: compact.diff Original Estimate: 72h Remaining Estimate: 72h Here is my case. After repeatedly minor compaction, the size of storefile is very large. Compaction with large storefile will waste much bandwidth, so i use the “hbase.hstore.compaction.max.size” to skip this case. But after use this config, i find that major compaction will be skipped forever when i read the source code and the deletes and muti-versions data my waste storage. So i had to modify the code. Now i'm try to submit my patch.But my patch is not perfect. I think there should be an other config to determine if the large size storefile should join major compaction in HColumnDescriptor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251939#comment-14251939 ] Hudson commented on HBASE-12223: FAILURE: Integrated in HBase-1.0 #600 (See [https://builds.apache.org/job/HBase-1.0/600/]) HBASE-12223 MultiTableInputFormatBase.getSplits is too slow (Yuanbo Peng) (tedyu: rev 9e18d8aa57051334b29d243af49f2a56be4bf078) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12720) Make InternalScan Public
Vladimir Rodionov created HBASE-12720: - Summary: Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Attachment: HBase-12720.patch Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Status: Patch Available (was: Open) The above patch is for master and all 0.98+. Should work. 0.94 patch will follow. Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12718) Convert TestAcidGuarantees from a unit test to an integration test
[
https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251948#comment-14251948
]
Hadoop QA commented on HBASE-12718:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688052/hbase-12718.patch
against master branch at commit 15cf0a6e7b10882f7fd205b65c2ef265a690597d.
ATTACHMENT ID: 12688052
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12136//console
This message is automatically generated.
Convert TestAcidGuarantees from a unit test to an integration test
--
Key: HBASE-12718
URL: https://issues.apache.org/jira/browse/HBASE-12718
Project: HBase
Issue Type: Bug
Components: hbase, integration tests, test
Affects Versions: 1.0.0, 2.0.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-12718.patch
TestAcidGurantees has a main method so that it can be executed via the
commandline. In the past this was run and we'd use external tools to inject
faults while it executed for an extended period of time. We've had the IT
framework we'd like to use the ChaosMonkey automation with it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Attachment: HBase-12720-0.94.patch 0.94 patch. Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch, HBase-12720.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Attachment: (was: HBase-12720.patch) Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Status: Open (was: Patch Available) Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12720) Make InternalScan Public
[
https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251961#comment-14251961
]
Hadoop QA commented on HBASE-12720:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688075/HBase-12720-0.94.patch
against master branch at commit 15cf0a6e7b10882f7fd205b65c2ef265a690597d.
ATTACHMENT ID: 12688075
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12138//console
This message is automatically generated.
Make InternalScan Public
Key: HBASE-12720
URL: https://issues.apache.org/jira/browse/HBASE-12720
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.25, 0.98.9
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10
Attachments: HBase-12720-0.94.patch
This is the request from sophisticated users :)
Rationale: We would like the internal scan to be made available so we can see
what is just in the MemStore (or in store files only).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Attachment: HBase-12720.patch.2 Small typo fixed. Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch, HBase-12720.patch.2 This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Status: Patch Available (was: Open) Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch, HBase-12720.patch.2 This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12012) Improve cancellation for the scan RPCs
[
https://issues.apache.org/jira/browse/HBASE-12012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251966#comment-14251966
]
Nick Dimiduk commented on HBASE-12012:
--
{quote}
bq. Separate interface? Can we not push this feature down into all
RetryingCallables?
This makes sense for the replica callables.. So it's done for those only.
{quote}
I suppose so. Proper interrupt handling in client RPC is something to address
separately.
bq. I don't think we need to add synchronized here. Can you see a reason for
this?
I was thinking another thread could access the instance, check it's
{{isCanceled}}, take some action even though the request hasn't actually be
canceled yet. Other than defensive coding, I don't have a specific problematic
interaction in mind.
lgtm, +1
Improve cancellation for the scan RPCs
--
Key: HBASE-12012
URL: https://issues.apache.org/jira/browse/HBASE-12012
Project: HBase
Issue Type: Sub-task
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 2.0.0, 1.1.0
Attachments: 12012-1.txt, 12012-2.txt, 12012-3.txt
Similar to HBASE-11564 but for scans.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12721) Extend hbase_docker to handle more useful use cases
Dima Spivak created HBASE-12721: --- Summary: Extend hbase_docker to handle more useful use cases Key: HBASE-12721 URL: https://issues.apache.org/jira/browse/HBASE-12721 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak Some simple work on using HBase with Docker was committed into /dev-support as hbase_docker; all this did was stand up a standalone cluster from source and start a shell. Now seems like a good time to extend this to be useful for applications that could actual benefit the community, especially around testing. Some ideas: - Integration testing would be much more accessible if people could stand up distributed HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite against it. - Binary compatibility testing of an HBase client is easiest when standing up an HBase cluster can be done once and then different client source/binary permutations run against it. - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache infrastructure. This proposal, whether it stays under /dev-support or moves out into it's own top-level module (hbase-docker would conveniently fit the existing schema :-)), strives to create a simple framework for deploying distributed, multi-container Apache HBase clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Extend hbase_docker to handle more useful use cases
[ https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251987#comment-14251987 ] stack commented on HBASE-12721: --- amen Extend hbase_docker to handle more useful use cases --- Key: HBASE-12721 URL: https://issues.apache.org/jira/browse/HBASE-12721 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak Some simple work on using HBase with Docker was committed into /dev-support as hbase_docker; all this did was stand up a standalone cluster from source and start a shell. Now seems like a good time to extend this to be useful for applications that could actual benefit the community, especially around testing. Some ideas: - Integration testing would be much more accessible if people could stand up distributed HBase clusters on a single host machine in a couple minutes and run our awesome hbase-it suite against it. - Binary compatibility testing of an HBase client is easiest when standing up an HBase cluster can be done once and then different client source/binary permutations run against it. - Upgrade testing, and especially rolling upgrade testing, doesn't have any upstream automation on build.apache.org, in part because it's a pain to set up x-node clusters on Apache infrastructure. This proposal, whether it stays under /dev-support or moves out into it's own top-level module (hbase-docker would conveniently fit the existing schema :-)), strives to create a simple framework for deploying distributed, multi-container Apache HBase clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12715) Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName
[
https://issues.apache.org/jira/browse/HBASE-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14251993#comment-14251993
]
Ted Yu commented on HBASE-12715:
Instead of using encodedRegionName when doing regionServerReport, can we make
use of the following public method of HRegionInfo ?
{code}
public static String encodeRegionName(final byte [] regionName) {
{code}
Should pass regionName to HRegionServer.getLastSequenceId instead of
encodedRegionName
--
Key: HBASE-12715
URL: https://issues.apache.org/jira/browse/HBASE-12715
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.98.9
Reporter: zhangduo
When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it
is renamed in master branch, doesn't matter) always returns -1, then I found
that we use encodedRegionName when calling this method. But when we doing
regionServerReport, we use regionName to report to HMaster(see the code in
HRegionServer.createRegionLoad).
We need to fix this, as seems we will disable distributed log replay in 0.98
by default?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12715) Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName
[ https://issues.apache.org/jira/browse/HBASE-12715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252006#comment-14252006 ] stack commented on HBASE-12715: --- [~Apache9] Please outline the implications (we are using region name comparing in some places but we are using encoded region name otherwise)? Generally, encoded region name -- because it is short -- should be used especially if a human does not have to read it. Should pass regionName to HRegionServer.getLastSequenceId instead of encodedRegionName -- Key: HBASE-12715 URL: https://issues.apache.org/jira/browse/HBASE-12715 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.98.9 Reporter: zhangduo When addressing HBASE-12405 I found that LastSequenceId.getLastSequenceId(it is renamed in master branch, doesn't matter) always returns -1, then I found that we use encodedRegionName when calling this method. But when we doing regionServerReport, we use regionName to report to HMaster(see the code in HRegionServer.createRegionLoad). We need to fix this, as seems we will disable distributed log replay in 0.98 by default? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12012) Improve cancellation for the scan RPCs
[ https://issues.apache.org/jira/browse/HBASE-12012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252011#comment-14252011 ] Devaraj Das commented on HBASE-12012: - Thanks, [~ndimiduk]. Plan to commit it in a few hours. Improve cancellation for the scan RPCs -- Key: HBASE-12012 URL: https://issues.apache.org/jira/browse/HBASE-12012 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 2.0.0, 1.1.0 Attachments: 12012-1.txt, 12012-2.txt, 12012-3.txt Similar to HBASE-11564 but for scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12012) Improve cancellation for the scan RPCs
[ https://issues.apache.org/jira/browse/HBASE-12012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252021#comment-14252021 ] stack commented on HBASE-12012: --- An interface called CancellableCallable that does not have 'cancel' in it is odd to me. It has 24public void startCancel(); When I see that, I start looking for isCancelled and cancelled methods, neither of which are here. In fact, why does this Interface have to be a Callable at all? Why not just have a Cancel Interface and decorate RetryingCallable with it? That is as far as I got. Improve cancellation for the scan RPCs -- Key: HBASE-12012 URL: https://issues.apache.org/jira/browse/HBASE-12012 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 2.0.0, 1.1.0 Attachments: 12012-1.txt, 12012-2.txt, 12012-3.txt Similar to HBASE-11564 but for scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252027#comment-14252027 ] stack commented on HBASE-12223: --- Is the failure related https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/600/ ? MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12223: --- Fix Version/s: (was: 1.0.0) MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252048#comment-14252048 ] Ted Yu commented on HBASE-12223: I took it out of branch-1.0 already MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252041#comment-14252041 ] Enis Soztutar commented on HBASE-12223: --- This was committed to branch-1.0 since Ted did not see my mail on the dev list. Let me run the test again and do a quick review. Otherwise, I'll revert the patch from 1.0.0 MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12722) Backport TestMultiTableInputFormat to 0.98 branch
Ted Yu created HBASE-12722: -- Summary: Backport TestMultiTableInputFormat to 0.98 branch Key: HBASE-12722 URL: https://issues.apache.org/jira/browse/HBASE-12722 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.98.10 When working on HBASE-12223, I noticed that TestMultiTableInputFormat was not in 0.98 branch. This issue is to backport the test to 0.98 branch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12722) Backport TestMultiTableInputFormat to 0.98 branch
[ https://issues.apache.org/jira/browse/HBASE-12722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12722: --- Attachment: 12722-0.98.txt Backport TestMultiTableInputFormat to 0.98 branch - Key: HBASE-12722 URL: https://issues.apache.org/jira/browse/HBASE-12722 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.98.10 Attachments: 12722-0.98.txt When working on HBASE-12223, I noticed that TestMultiTableInputFormat was not in 0.98 branch. This issue is to backport the test to 0.98 branch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12722) Backport TestMultiTableInputFormat to 0.98 branch
[ https://issues.apache.org/jira/browse/HBASE-12722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252068#comment-14252068 ] Ted Yu commented on HBASE-12722: The test passed locally. [~apurtell]: Do you want to take a look ? Backport TestMultiTableInputFormat to 0.98 branch - Key: HBASE-12722 URL: https://issues.apache.org/jira/browse/HBASE-12722 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.98.10 Attachments: 12722-0.98.txt When working on HBASE-12223, I noticed that TestMultiTableInputFormat was not in 0.98 branch. This issue is to backport the test to 0.98 branch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12722) Backport TestMultiTableInputFormat to 0.98 branch
[ https://issues.apache.org/jira/browse/HBASE-12722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12722: --- Status: Patch Available (was: Open) Backport TestMultiTableInputFormat to 0.98 branch - Key: HBASE-12722 URL: https://issues.apache.org/jira/browse/HBASE-12722 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.98.10 Attachments: 12722-0.98.txt When working on HBASE-12223, I noticed that TestMultiTableInputFormat was not in 0.98 branch. This issue is to backport the test to 0.98 branch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252072#comment-14252072
]
stack commented on HBASE-12684:
---
Here is a quick review. Hope it helps some.
Why out of interest the below change?
830 return (value != null)? Integer.valueOf(value).intValue():
DEFAULT_TTL; 830 return (value != null) ? Integer.parseInt(value) :
DEFAULT_TTL;
The below is not a repeat because we are replacing old rpc with this new one?
81private static final byte[] MAGIC = new byte[] { 'H', 'B', 'a', 's' };
This looks great: 201
f.channel().pipeline().addFirst(saslHandler);
This change is good too:
150 extends AbstractRpcClient.BlockingRpcChannelImplementation {
149 extends RpcClientImpl.BlockingRpcChannelImplementation {
Patch looks great. On the tests that are zombies, they are probably missing
(timeout=XX) annotations after @Test. Feel free to add if it will help you
debug. When you get a chance, make a list of what you will be able to remove
after this is all working. I love removing code.
Thanks [~jurmous]
Add new AsyncRpcClient
--
Key: HBASE-12684
URL: https://issues.apache.org/jira/browse/HBASE-12684
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12684-v1.patch, HBASE-12684-v2.patch,
HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch,
HBASE-12684.patch
With the changes in HBASE-12597 it is possible to add new RpcClients. This
issue is about adding a new Async RpcClient which would enable HBase to do
non blocking protobuf service communication.
Besides delivering a new AsyncRpcClient I would also like to ask the question
what it would take to replace the current RpcClient? This would enable to
simplify async code in some next issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252084#comment-14252084 ] Virag Kothari commented on HBASE-12694: --- Test also passes for me. Thanks Vandana. Committing this in a short time testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition - Key: HBASE-12694 URL: https://issues.apache.org/jira/browse/HBASE-12694 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 0.98.9 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Attachments: HBASE-12694-98.1.patch, HBASE-12694-branch-1.1.patch, HBASE-12694-branch-1.patch, HBASE-12694-master.patch There seems to a clean up issue with the unit test testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class. It always has a region in transition. So any unit test that executes after that test, will cause balance method to return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12694: -- Status: Patch Available (was: Open) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition - Key: HBASE-12694 URL: https://issues.apache.org/jira/browse/HBASE-12694 Project: HBase Issue Type: Bug Affects Versions: 0.98.9, 1.0.0 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Attachments: HBASE-12694-98.1.patch, HBASE-12694-branch-1.1.patch, HBASE-12694-branch-1.patch, HBASE-12694-master.patch There seems to a clean up issue with the unit test testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class. It always has a region in transition. So any unit test that executes after that test, will cause balance method to return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252088#comment-14252088 ] stack commented on HBASE-12694: --- +1 testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition - Key: HBASE-12694 URL: https://issues.apache.org/jira/browse/HBASE-12694 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 0.98.9 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Attachments: HBASE-12694-98.1.patch, HBASE-12694-branch-1.1.patch, HBASE-12694-branch-1.patch, HBASE-12694-master.patch There seems to a clean up issue with the unit test testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class. It always has a region in transition. So any unit test that executes after that test, will cause balance method to return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12711) Fix new findbugs warnings in hbase-thrift module
[ https://issues.apache.org/jira/browse/HBASE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-12711: Attachment: HBASE-12711.patch Attaching the first cut. Let's see how it goes. Fix new findbugs warnings in hbase-thrift module Key: HBASE-12711 URL: https://issues.apache.org/jira/browse/HBASE-12711 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12711.patch From https://builds.apache.org/job/PreCommit-HBASE-Build/12121/artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html , there were 5 findbugs warnings introduced. This issue fixes the new warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12711) Fix new findbugs warnings in hbase-thrift module
[ https://issues.apache.org/jira/browse/HBASE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-12711: Status: Patch Available (was: Open) Fix new findbugs warnings in hbase-thrift module Key: HBASE-12711 URL: https://issues.apache.org/jira/browse/HBASE-12711 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12711.patch From https://builds.apache.org/job/PreCommit-HBASE-Build/12121/artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html , there were 5 findbugs warnings introduced. This issue fixes the new warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12012) Improve cancellation for the scan RPCs
[ https://issues.apache.org/jira/browse/HBASE-12012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252099#comment-14252099 ] Devaraj Das commented on HBASE-12012: - [~stack], thanks for review. I don't have the isCancelled method in the interface since it was not needed yet. But I can add it if you like for completeness. Improve cancellation for the scan RPCs -- Key: HBASE-12012 URL: https://issues.apache.org/jira/browse/HBASE-12012 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 2.0.0, 1.1.0 Attachments: 12012-1.txt, 12012-2.txt, 12012-3.txt Similar to HBASE-11564 but for scans. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[
https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252105#comment-14252105
]
Hadoop QA commented on HBASE-12694:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12687840/HBASE-12694-98.1.patch
against master branch at commit 9ae615f82bc0521b7301bb5ea99f0367fc20f746.
ATTACHMENT ID: 12687840
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12141//console
This message is automatically generated.
testTableExistsIfTheSpecifiedTableRegionIsSplitParent in
TestSplitTransactionOnCluster class leaves regions in transition
-
Key: HBASE-12694
URL: https://issues.apache.org/jira/browse/HBASE-12694
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.9
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: HBASE-12694-98.1.patch, HBASE-12694-branch-1.1.patch,
HBASE-12694-branch-1.patch, HBASE-12694-master.patch
There seems to a clean up issue with the unit test
testTableExistsIfTheSpecifiedTableRegionIsSplitParent in
TestSplitTransactionOnCluster class. It always has a region in transition. So
any unit test that executes after that test, will cause balance method to
return false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12694) testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition
[ https://issues.apache.org/jira/browse/HBASE-12694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virag Kothari updated HBASE-12694: -- Attachment: HBASE-12694-master.patch Forgot to let the pre-commit build run. Reverting and reattaching the patch testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class leaves regions in transition - Key: HBASE-12694 URL: https://issues.apache.org/jira/browse/HBASE-12694 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 0.98.9 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Attachments: HBASE-12694-98.1.patch, HBASE-12694-branch-1.1.patch, HBASE-12694-branch-1.patch, HBASE-12694-master.patch, HBASE-12694-master.patch There seems to a clean up issue with the unit test testTableExistsIfTheSpecifiedTableRegionIsSplitParent in TestSplitTransactionOnCluster class. It always has a region in transition. So any unit test that executes after that test, will cause balance method to return false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252111#comment-14252111
]
Hadoop QA commented on HBASE-12684:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688068/HBASE-12684-v5.patch
against master branch at commit 15cf0a6e7b10882f7fd205b65c2ef265a690597d.
ATTACHMENT ID: 12688068
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:red}-1 javac{color}. The applied patch generated 112 javac compiler
warnings (more than the master's current 111 warnings).
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2088 checkstyle errors (more than the master's current 2087 errors).
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.master.TestRestartCluster
org.apache.hadoop.hbase.master.handler.TestCreateTableHandler
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.client.TestHCM.testClusterConnection(TestHCM.java:165)
at
org.apache.camel.test.junit4.CamelTestSupport.doStopCamelContext(CamelTestSupport.java:450)
at
org.apache.camel.test.junit4.CamelTestSupport.tearDown(CamelTestSupport.java:351)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12137//console
This message is automatically generated.
Add new AsyncRpcClient
--
Key: HBASE-12684
URL: https://issues.apache.org/jira/browse/HBASE-12684
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12684-v1.patch, HBASE-12684-v2.patch,
HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch,
HBASE-12684.patch
With the changes in HBASE-12597 it is possible to add new RpcClients. This
issue is about adding a new Async RpcClient which would enable HBase to do
non blocking protobuf service communication.
Besides delivering a new AsyncRpcClient I would also like to ask the question
what it would take to replace
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Attachment: (was: HBase-12720.patch.2) Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Status: Open (was: Patch Available) Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch, HBase-12720.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Attachment: HBase-12720.patch Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.94.25, 0.98.9 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch, HBase-12720.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252134#comment-14252134 ] Virag Kothari commented on HBASE-11290: --- I should be able to use IdLock. Let me try refactoring. Unlock RegionStates --- Key: HBASE-11290 URL: https://issues.apache.org/jira/browse/HBASE-11290 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Virag Kothari Fix For: 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, HBASE-11290.draft.patch Even though RegionStates is a highly accessed data structure in HMaster. Most of it's methods are synchronized. Which limits concurrency. Even simply making some of the getters non-synchronized by using concurrent data structures has helped with region assignments. We can go as simple as this approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12720) Make InternalScan Public
[ https://issues.apache.org/jira/browse/HBASE-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-12720: -- Status: Patch Available (was: Open) Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 Attachments: HBase-12720-0.94.patch, HBase-12720.patch This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252171#comment-14252171
]
James Taylor commented on HBASE-12566:
--
What does it mean if an interface is marked as @InterfaceAudience.Private? An
example is RegionScanner which is returned from the doPostScannerOpen
RegionObserver coprocessors method. Phoenix uses this interface to provide a
different scanner that does aggregation from a coprocessor hook. Is this
forbidden as well?
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Labels: Phoenix
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252185#comment-14252185 ] Hudson commented on HBASE-12223: SUCCESS: Integrated in HBase-1.0 #601 (See [https://builds.apache.org/job/HBase-1.0/601/]) HBASE-12223 Revert from 1.0 branch (tedyu: rev c86607fd71a86904592a0c1789872fd40e2dafdb) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: HBASE-12223-0.94.patch, HBASE-12223-0.98.patch, HBASE-12223-v1.patch, HBASE-12223-v2.patch, HBASE-12223-v3.patch, HBASE-12223-v4.patch, HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252207#comment-14252207
]
Andrew Purtell commented on HBASE-12566:
Yes, but in that case this type is used by the coprocessor API so we need to
change the annotation or provide a replacement copacetic type in 1.0.
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Labels: Phoenix
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
