[jira] [Updated] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12684: -- Attachment: requests.png Rough illustration of the difference. On left is the heap allocated async client and on right is the old client. I haven't looked at the two clients in a profiler.. perhaps the friction will be more obvious there? Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.1.patch, HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch, requests.png With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265746#comment-14265746 ] Hudson commented on HBASE-12779: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #742 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/742/]) HBASE-12779 SplitTransaction: Add metrics. (larsh: rev 652da5404336e15064e12e2e278c68985ec25a3a) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitRequest.java * hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12811) NPE while scan a table with user associated with multiple groups.
Ashish Singhi created HBASE-12811:
-
Summary: NPE while scan a table with user associated with multiple
groups.
Key: HBASE-12811
URL: https://issues.apache.org/jira/browse/HBASE-12811
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
A user is associated with two groups.
{noformat}
/hbase/bin groups ashish_test
ashish_test : defaultgroup ashish_test_1420524824527
{noformat}
One of its group is granted permission on a table as shown by user_permission
command.
{noformat}
hbase(main):005:0 user_permission 't1'
User
Table,Family,Qualifier:Permission
@ashish_test_1420524824527 t1,,: [Permission:
actions=EXEC,WRITE,CREATE]
@ashish_test_1420524824527 t1,d,: [Permission:
actions=EXEC,WRITE,CREATE]
hbase t1,,: [Permission:
actions=READ,WRITE,EXEC,CREATE,ADMIN]
3 row(s) in 0.3710 seconds
{noformat}
Now when this user try the scan the table, we get the following exception.
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hbase.security.access.TablePermission.implies(TablePermission.java:215)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:340)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:332)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorizeGroup(TableAuthManager.java:473)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:490)
at
org.apache.hadoop.hbase.security.access.TableAuthManager.authorize(TableAuthManager.java:500)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:415)
at
org.apache.hadoop.hbase.security.access.AccessController.permissionGranted(AccessController.java:484)
at
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1504)
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:2027)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1987)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3102)
{noformat}
*Note:* Line numbers may not match.
Exception is coming because the other group of same user which has not been
granted permission on the table will have the TablePermission's table(name) as
null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12771) TestFailFast#testFastFail failing
[
https://issues.apache.org/jira/browse/HBASE-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265756#comment-14265756
]
Hudson commented on HBASE-12771:
FAILURE: Integrated in HBase-1.1 #59 (See
[https://builds.apache.org/job/HBase-1.1/59/])
HBASE-12771 TestFailFast#testFastFail failing (Manukranth Kolloju) (stack: rev
6fcc8d7c3e00545b795f8668040aa6db373ded8b)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
TestFailFast#testFastFail failing
-
Key: HBASE-12771
URL: https://issues.apache.org/jira/browse/HBASE-12771
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: Manukranth Kolloju
Fix For: 1.0.0, 1.1.0
Attachments: HBASE-12771-v1.patch, HBASE-12771-v2.patch
Fails on our internal rig and from time to time on apache. Here is latest:
{code}
org.apache.hadoop.hbase.client.TestFastFail.testFastFail
Failing for the past 1 build (Since Failed#654 )
Took 7.1 sec.
Error Message
Only few thread should ideally be waiting for the dead regionserver to be
coming back. numBlockedWorkers:155 threads that retried : 10
Stacktrace
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:155 threads that
retried : 10
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:270)
{code}
Opening this issue so can start tracking the fails. Looking in log, nothing
obvious. Will be back to this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12771) TestFailFast#testFastFail failing
[
https://issues.apache.org/jira/browse/HBASE-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265768#comment-14265768
]
Hudson commented on HBASE-12771:
SUCCESS: Integrated in HBase-1.0 #637 (See
[https://builds.apache.org/job/HBase-1.0/637/])
HBASE-12771 TestFailFast#testFastFail failing (Manukranth Kolloju) (stack: rev
a5cfd4953d71e244742e9a90bbe2171b56262d7c)
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
TestFailFast#testFastFail failing
-
Key: HBASE-12771
URL: https://issues.apache.org/jira/browse/HBASE-12771
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: Manukranth Kolloju
Fix For: 1.0.0, 1.1.0
Attachments: HBASE-12771-v1.patch, HBASE-12771-v2.patch
Fails on our internal rig and from time to time on apache. Here is latest:
{code}
org.apache.hadoop.hbase.client.TestFastFail.testFastFail
Failing for the past 1 build (Since Failed#654 )
Took 7.1 sec.
Error Message
Only few thread should ideally be waiting for the dead regionserver to be
coming back. numBlockedWorkers:155 threads that retried : 10
Stacktrace
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:155 threads that
retried : 10
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:270)
{code}
Opening this issue so can start tracking the fails. Looking in log, nothing
obvious. Will be back to this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12769) Replication fails to delete all corresponding zk nodes when peer is removed
[ https://issues.apache.org/jira/browse/HBASE-12769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264324#comment-14264324 ] cuijianwei commented on HBASE-12769: I tried to fix the problem following the above steps, it seems many places of current implementation need to be changed if adding a REMOVING peer state, such as we must avoid enable/disable a peer in REMOVING state, the PeersWatcher should also be changed to know this state, etc. We must be careful to keep every place right after adding REMOVING state. Maybe, we can try another way to solve the problem: 1. The logic of removing peer won't change, it is also the regionserver's responsibility to delete hlog queues of removed peer; 2. When adding a new peer, we can firstly check all replication rs znode and throw exception if there are uncleaned queues belongs to the peer, this will prevent adding a new peer before old queues cleaned; 3. Teach HBCK to discover the uncleaned queues for removed peer and fix it. How do you think about these steps? [~apurtell] Replication fails to delete all corresponding zk nodes when peer is removed --- Key: HBASE-12769 URL: https://issues.apache.org/jira/browse/HBASE-12769 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.99.2 Reporter: cuijianwei Priority: Minor When removing a peer, the client side will delete peerId under peersZNode node; then alive region servers will be notified and delete corresponding hlog queues under its rsZNode of replication. However, if there are failed servers whose hlog queues have not been transferred by alive servers(this likely happens if setting a big value to replication.sleep.before.failover and lots of region servers restarted), these hlog queues won't be deleted after the peer is removed. I think remove_peer should guarantee all corresponding zk nodes have been removed after it completes; otherwise, if we create a new peer with the same peerId with the removed one, there might be unexpected data to be replicated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12804) ImportTsv fails to delete partition files created by it
[
https://issues.apache.org/jira/browse/HBASE-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264343#comment-14264343
]
Hadoop QA commented on HBASE-12804:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690016/HBASE-12804.patch
against master branch at commit ac95cc1fbb951bb9db96f2738f621d1d7cd45739.
ATTACHMENT ID: 12690016
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12300//console
This message is automatically generated.
ImportTsv fails to delete partition files created by it
---
Key: HBASE-12804
URL: https://issues.apache.org/jira/browse/HBASE-12804
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0
Attachments: HBASE-12804.patch
{code}
fs.deleteOnExit(partitionsPath);
writePartitions(job.getConfiguration(), partitionsPath, splitPoints);
{code}
In the above code deleteOnExit will return without adding file in the
deleteOnExit set as the file is not created till then.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Attachment: HBASE-12796-v3.patch - Fixed test fail issues. [~te...@apache.org] Are you sure you are on master? I didn't need to update the patch for it to work. Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch, HBASE-12796-v3.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[
https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264399#comment-14264399
]
Hadoop QA commented on HBASE-12796:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690053/HBASE-12796-v3.patch
against master branch at commit ac95cc1fbb951bb9db96f2738f621d1d7cd45739.
ATTACHMENT ID: 12690053
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 555
new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12301//console
This message is automatically generated.
Clean up HTable and HBaseAdmin deprecated constructor usage
---
Key: HBASE-12796
URL: https://issues.apache.org/jira/browse/HBASE-12796
Project: HBase
Issue Type: Improvement
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch,
HBASE-12796-v3.patch, HBASE-12796.patch
The are a lot of calls to the deprecated calls to HBaseAdmin and HTable
constructors in many tests and even some in production code.
This issue is about removing most of them so most code does not create
redundant Connections and underlying pools and RPC clients. Hopefully making
the tests more lightweight.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Attachment: HBASE-12796-v4.patch It seemed git didn't want to pull the latest changes from apache git mirror so I switched to the github mirror and merged the conflicts. Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch, HBASE-12796-v3.patch, HBASE-12796-v4.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[
https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264539#comment-14264539
]
Hadoop QA commented on HBASE-12796:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690060/HBASE-12796-v4.patch
against master branch at commit ac95cc1fbb951bb9db96f2738f621d1d7cd45739.
ATTACHMENT ID: 12690060
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 544
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12302//console
This message is automatically generated.
Clean up HTable and HBaseAdmin deprecated constructor usage
---
Key: HBASE-12796
URL: https://issues.apache.org/jira/browse/HBASE-12796
Project: HBase
Issue Type: Improvement
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch,
HBASE-12796-v3.patch, HBASE-12796-v4.patch, HBASE-12796.patch
The are a lot of calls to the deprecated calls to HBaseAdmin and HTable
constructors in many tests and even some in production code.
This issue is about removing most of them so most code does not create
redundant Connections and underlying pools and RPC clients. Hopefully making
the tests more lightweight.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Attachment: HBASE-12796-v5.patch Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch, HBASE-12796-v3.patch, HBASE-12796-v4.patch, HBASE-12796-v5.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[
https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264609#comment-14264609
]
Hadoop QA commented on HBASE-12796:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690071/HBASE-12796-v5.patch
against master branch at commit ac95cc1fbb951bb9db96f2738f621d1d7cd45739.
ATTACHMENT ID: 12690071
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 544
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.master.TestRestartCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12303//console
This message is automatically generated.
Clean up HTable and HBaseAdmin deprecated constructor usage
---
Key: HBASE-12796
URL: https://issues.apache.org/jira/browse/HBASE-12796
Project: HBase
Issue Type: Improvement
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch,
HBASE-12796-v3.patch, HBASE-12796-v4.patch, HBASE-12796-v5.patch,
HBASE-12796.patch
The are a lot of calls to the deprecated calls to HBaseAdmin and HTable
constructors in many tests and even some in production code.
This issue is about removing most of them so most code does not create
redundant Connections and underlying pools and RPC clients. Hopefully making
the tests more lightweight.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264551#comment-14264551 ] Jurriaan Mous commented on HBASE-12684: --- [~stack] Any updates on the performance tests? :) Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12804) ImportTsv fails to delete partition files created by it
[
https://issues.apache.org/jira/browse/HBASE-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264777#comment-14264777
]
Ashish Singhi commented on HBASE-12804:
---
I don't have code to see now.
But what I have observed is deleteOnExit will add to the set only when the path
already exists.
writePartitions will be one which will create the file and then once the path
exists we should call deleteOnExit so that we can add the file to the
deleteOnExit set.
ImportTsv fails to delete partition files created by it
---
Key: HBASE-12804
URL: https://issues.apache.org/jira/browse/HBASE-12804
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0
Attachments: HBASE-12804.patch
{code}
fs.deleteOnExit(partitionsPath);
writePartitions(job.getConfiguration(), partitionsPath, splitPoints);
{code}
In the above code deleteOnExit will return without adding file in the
deleteOnExit set as the file is not created till then.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12804) ImportTsv fails to delete partition files created by it
[
https://issues.apache.org/jira/browse/HBASE-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264779#comment-14264779
]
Ted Yu commented on HBASE-12804:
+1
ImportTsv fails to delete partition files created by it
---
Key: HBASE-12804
URL: https://issues.apache.org/jira/browse/HBASE-12804
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0
Attachments: HBASE-12804.patch
{code}
fs.deleteOnExit(partitionsPath);
writePartitions(job.getConfiguration(), partitionsPath, splitPoints);
{code}
In the above code deleteOnExit will return without adding file in the
deleteOnExit set as the file is not created till then.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12804) ImportTsv fails to delete partition files created by it
[
https://issues.apache.org/jira/browse/HBASE-12804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12804:
---
Resolution: Fixed
Fix Version/s: (was: 1.0.1)
1.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the patch, Ashish.
ImportTsv fails to delete partition files created by it
---
Key: HBASE-12804
URL: https://issues.apache.org/jira/browse/HBASE-12804
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.9
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12804.patch
{code}
fs.deleteOnExit(partitionsPath);
writePartitions(job.getConfiguration(), partitionsPath, splitPoints);
{code}
In the above code deleteOnExit will return without adding file in the
deleteOnExit set as the file is not created till then.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12791) HBase does not attempt to clean up an aborted split when the regionserver shutting down
[
https://issues.apache.org/jira/browse/HBASE-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265160#comment-14265160
]
Ted Yu commented on HBASE-12791:
+1
HBase does not attempt to clean up an aborted split when the regionserver
shutting down
---
Key: HBASE-12791
URL: https://issues.apache.org/jira/browse/HBASE-12791
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.0
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.0.1
Attachments: HBASE-12791.patch, HBASE-12791_v2.patch
HBase not cleaning the daughter region directories from HDFS if region
server shut down after creating the daughter region directories during the
split.
Here the logs.
- RS shutdown after creating the daughter regions.
{code}
2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385]
zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to
RS_ZK_REQUEST_REGION_SPLIT
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closing
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.: disabling compactions
flushes
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Updates disabled for region
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:41,516 INFO
[StoreCloserThread-t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.-1]
regionserver.HStore: Closed f
2014-12-31 09:05:41,518 INFO [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closed
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t dd9731ee43b104da565257ca1539aa8c
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,,1419996941401.dd9731ee43b104da565257ca1539aa8c.
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t 2e40a44511c0e187d357d651f13a1dab
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,row2,1419996941401.2e40a44511c0e187d357d651f13a1dab.
Wed Dec 31 09:06:30 IST 2014 Terminating regionserver
2014-12-31 09:06:30,465 INFO [Thread-8] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@42d2282e
{code}
- Skipping rollback if RS stopped or stopping so we end up in dirty daughter
regions in HDFS.
{code}
2014-12-31 09:07:49,547 INFO [regionserver60020-splits-1419996941385]
regionserver.SplitRequest: Skip rollback/cleanup of failed split of
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f. because server is stopped
java.io.InterruptedIOException: Interrupted after 0 tries on 350
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:156)
{code}
Because of this hbck always showing inconsistencies.
{code}
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/2e40a44511c0e187d357d651f13a1dab,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/dd9731ee43b104da565257ca1539aa8c,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
{code}
If we try to repair then we end up in overlap regions in hbase:meta. and both
daughter regions and parent are online.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12745) Visibility Labels: support visibility labels for user groups.
[
https://issues.apache.org/jira/browse/HBASE-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-12745:
-
Attachment: HBASE-12745-master-v3.patch
Attached master-v3 that addressed the long lines.
Visibility Labels: support visibility labels for user groups.
--
Key: HBASE-12745
URL: https://issues.apache.org/jira/browse/HBASE-12745
Project: HBase
Issue Type: Improvement
Components: security
Affects Versions: 1.0.0, 0.98.9, 0.99.2
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0
Attachments: HBASE-12745-master-v1.patch,
HBASE-12745-master-v2.patch, HBASE-12745-master-v3.patch
The thinking is that we should support visibility labels to be associated
with user groups.
We will then be able grant visibility labels to a group in addition to
individual users, which provides convenience and usability.
We will use '@group' to denote a group name, as similarly done in
AcccessController.
For example,
{code}
set_auths '@group1', ['SECRET','PRIVATE']
{code}
{code}
get_auth '@group1'
{code}
A user belonging to 'group1' will have all the visibility labels granted to
'group1'
We'll also support super user groups as specified in hbase-site.xml.
The code update will mainly be on the server side VisibilityLabelService
implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12785: --- Resolution: Fixed Fix Version/s: 1.1.0 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Addressed above comment on commit. Thanks for the review, Sean. Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: 12785-001.patch, 12785-002.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265216#comment-14265216 ] Dima Spivak commented on HBASE-12806: - Or even easier: just use the createTable(TableName tableName, byte[] family), since in this instance, the created table from TestHBaseFsck#testTableWithNoRegions only has TableName and a CF in the HTableDescriptor anyway. [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12808) Use Java API Compliance Checker for binary/source compatibility
Dima Spivak created HBASE-12808: --- Summary: Use Java API Compliance Checker for binary/source compatibility Key: HBASE-12808 URL: https://issues.apache.org/jira/browse/HBASE-12808 Project: HBase Issue Type: Improvement Components: test Reporter: Dima Spivak Assignee: Dima Spivak Following [~busbey]'s suggestion in HBASE-12556, I've spent some time playing with the [Java API Compliance Checker|http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker] and think it would be a great addition to /dev-support. I propose that we use it to replace the JDiff wrappers we currently have there (since it does what JDiff does and more), and look into putting up automation at builds.apache.org to run the tool regularly (e.g. latest release of a particular branch vs. latest commit of that same branch). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Solomon Duskis reassigned HBASE-12728:
--
Assignee: Solomon Duskis
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
Assignee: Solomon Duskis
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12809) Remove unnecessary calls to Table.setAutoFlush()
Solomon Duskis created HBASE-12809:
--
Summary: Remove unnecessary calls to Table.setAutoFlush()
Key: HBASE-12809
URL: https://issues.apache.org/jira/browse/HBASE-12809
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
It looks like there are a lot of places where setAutoFlushTo() is called in
places where that's not necessary. HBASE-12728 will likely result in removing
the flushCommits() method from Table. The patch for this issue should remove
all unnecessary calls to setAutoFlushTo() to prepare for the full fix.
setAutoFlushTo(true) is unnecessary on newly constructed HTables, since
autoFlush is true by default. Calls like the following
{code}
table.setAutoFlushTo(false);
for(...) {
Put put = new Put(...);
...
table.put(put);
}
table.flushCommits();
{code}
Is equivalent in functionality to:
{code}
ListPut puts = new ArrayList();
for(...) {
Put put = new Put(...);
...
puts.add(put);
}
table.put(puts);
{code}
The put(ListPut) semantics ought to be the preferred approach.
Note: here's the code for put(Put) and put(ListPut):
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265324#comment-14265324 ] Jurriaan Mous commented on HBASE-12684: --- Thanks for the performance test! bq. The async rpc client is running slower, about 15% less throughput (see below for more detail). This needs to be improved! bq. Async client is doing await on future/promise or we are allocating a direct buffer. The await is not something that can change with the current blocking usage and the sync and async client should both block in a comparable way. I think I have done all I can here but will see if anything can help here by moving some work to other threads in the Netty EventLoop. And it seems that the direct buffer disadvantages are working against us. I will look into this more and swap them with heap buffers were necessary. bq. You can't use a netty bytebuf pool? (Direct buffer allocation is slow) I thought I was using a pooled buffer by using channel.alloc() to get them. The documentation I read about it suggested a pool was the default allocator. But looking into what is happening in code I get an unpooled allocator. (As your stack trace also shows) This is certainly not good and I will switch this to a Netty PooledByteBufAllocator and configure it with some defaults the Netty experts are suggesting. I will return to this tomorrow! Again thanks! Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.1.patch, HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
[
https://issues.apache.org/jira/browse/HBASE-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265337#comment-14265337
]
Hadoop QA commented on HBASE-12793:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690171/0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l-v0.98.patch
against master branch at commit 5b53a187d26ac54b7cd2c3d4a011bcc02931de99.
ATTACHMENT ID: 12690171
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12315//console
This message is automatically generated.
[hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry
until hbase.hbck.close.timeout expires
-
Key: HBASE-12793
URL: https://issues.apache.org/jira/browse/HBASE-12793
Project: HBase
Issue Type: Sub-task
Components: hbck
Affects Versions: 1.0.0, 0.98.9
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Priority: Minor
Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0
Attachments:
0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch,
0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch,
0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l-v0.98.patch,
0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch
This is subtask on HBASE-12131 in order to handle gracefully network
partitions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12806: -- Affects Version/s: 2.0.0 1.0.0 Fix Version/s: 2.0.0 [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Fix For: 2.0.0 Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12806: -- Status: Patch Available (was: Open) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Fix For: 2.0.0 Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12802) Remove unnecessary Table.flushCommits()
[ https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12802: --- Attachment: HBASE-12802B.patch Fixed the previous broken test. It was related to passing an immutable List into delete(ListDelete). It turns out that delete(ListDelete) modifies the List. I thought I fixed all the tests before, but I guess I missed that one, since it failed for me as well when I ran it locally. Remove unnecessary Table.flushCommits() --- Key: HBASE-12802 URL: https://issues.apache.org/jira/browse/HBASE-12802 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12802.patch, HBASE-12802B.patch It looks like there are a lot of places where flushCommits() are called on tables that do not have autoFlush turned on. That might be a legacy concern from pre-autoFlush days. HBASE-12728 will likely result in removing the flushCommits() method from Table. The patch for this issue should remove all unnecessary calls to flushCommit() to prepare for the full fix.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[
https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265183#comment-14265183
]
Sean Busbey commented on HBASE-12785:
-
+1, minor issue:
{code}
+final int timeoutInSeconds = 30;
+ stream = futureTask.get(30, TimeUnit.SECONDS);
+} catch (ExecutionException ee) {
+ LOG.warn(Encountered exception when opening lock file, ee);
+} catch (InterruptedException ie) {
+ LOG.warn(Interrupted when opening lock file, ie);
+ Thread.currentThread().interrupt();
+} catch (TimeoutException exception) {
+ // took too long to obtain lock
+ LOG.warn(Took more than + timeoutInSeconds + seconds in obtaining
lock);
+ futureTask.cancel(true);
{code}
Either make futureTask.get use the timeoutInSeconds variable or skip it
entirely and hard-code 30 in the error message.
Use FutureTask to timeout the attempt to get the lock for hbck
--
Key: HBASE-12785
URL: https://issues.apache.org/jira/browse/HBASE-12785
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Attachments: 12785-001.patch, 12785-002.patch
In reviewing HBASE-12607, Sean pointed out:
It would be nice if we used a
[FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html]
to timeout the attempt to get the lock rather than wait the whole period and
then fail.
This issue is to address Sean's review comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-9117) Remove HTablePool and all HConnection pooling related APIs
[ https://issues.apache.org/jira/browse/HBASE-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265190#comment-14265190 ] Nick Dimiduk commented on HBASE-9117: - This ticket was more about managed vs unmanaged connections. HTablePool is indeed intended for removal, I guess that piece slipped through the cracks on this JIRA. There's more conversation over on HBASE-12728. Remove HTablePool and all HConnection pooling related APIs -- Key: HBASE-9117 URL: https://issues.apache.org/jira/browse/HBASE-9117 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Mikhail Antonov Priority: Critical Fix For: 2.0.0, 0.99.2 Attachments: HBASE-9117.00.patch, HBASE-9117.01.patch, HBASE-9117.02.patch, HBASE-9117.03.patch, HBASE-9117.04.patch, HBASE-9117.05.patch, HBASE-9117.06.patch The recommended way is now: # Create an HConnection: HConnectionManager.createConnection(...) # Create a light HTable: HConnection.getTable(...) # table.close() # connection.close() All other API and pooling will be removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265209#comment-14265209
]
Solomon Duskis commented on HBASE-12802:
[~stack]: I'll run the tests a few times. The premise here is that
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBufferr.
If I'd have to guess, I would say that flushCommits() used to affect all
mutations, but along the way, there were dramatic changes to how the various
mutation types are handled.
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265211#comment-14265211 ] Dima Spivak commented on HBASE-12806: - My favorite race condition! :) Just a thought, [~esteban]: since HBaseTestingUtility doesn't already have a createTable method that simply takes an HTD, want to add one instead of calling the (HTD, byte[][]) version with a null for the second argument? [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11983) HRegion constructors should not create HLog
[
https://issues.apache.org/jira/browse/HBASE-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265264#comment-14265264
]
Sean Busbey commented on HBASE-11983:
-
I'm happy to do my superset patch either here as a follow on jira. I probably
won't have time to pick up again until the end of htis week.
HRegion constructors should not create HLog
Key: HBASE-11983
URL: https://issues.apache.org/jira/browse/HBASE-11983
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Enis Soztutar
Assignee: Nick Dimiduk
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-11983.00.patch, HBASE-11983.01.patch
We should get rid of HRegion creating its own HLog. It should ALWAYS get the
log from outside.
I think this was added for unit tests, but we should refrain from such
practice in the future (adding UT constructors always leads to weird and
critical bugs down the road). See recent: HBASE-11982, HBASE-11654.
Get rid of weird things like ignoreHLog:
{code}
/**
* @param ignoreHLog - true to skip generate new hlog if it is null, mostly
for createTable
*/
public static HRegion createHRegion(final HRegionInfo info, final Path
rootDir,
final Configuration conf,
final HTableDescriptor hTableDescriptor,
final HLog hlog,
final boolean initialize, final boolean
ignoreHLog)
{code}
We can unify all the createXX and newXX methods and separate creating a
region in the file system vs opening a region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12806: -- Attachment: 0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch Thanks for the feedback [~dimaspivak] just changed to the existing method instead of passing the null splits. [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch, 0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265340#comment-14265340
]
Hadoop QA commented on HBASE-12806:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690151/0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch
against master branch at commit 24bcebdeeb5c8025a1171fd73b5f38e666ae819b.
ATTACHMENT ID: 12690151
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12309//console
This message is automatically generated.
[hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
--
Key: HBASE-12806
URL: https://issues.apache.org/jira/browse/HBASE-12806
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 1.0.0, 2.0.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Priority: Minor
Attachments:
0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch,
0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch
TestHBaseFsck should wait until all regions have been assigned after the
table has been created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12808) Use Java API Compliance Checker for binary/source compatibility
[ https://issues.apache.org/jira/browse/HBASE-12808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265234#comment-14265234 ] Dima Spivak commented on HBASE-12808: - Also, in terms of limiting the scans to only run on InterfaceAudience.Public classes, we can get the desired behavior by using reflection to generate a list of classes (which can be passed into Java ACC). Longer term, I'll reach out the developer and see if he'd be open to adding the feature/letting us add it as a PR. Use Java API Compliance Checker for binary/source compatibility --- Key: HBASE-12808 URL: https://issues.apache.org/jira/browse/HBASE-12808 Project: HBase Issue Type: Improvement Components: test Reporter: Dima Spivak Assignee: Dima Spivak Following [~busbey]'s suggestion in HBASE-12556, I've spent some time playing with the [Java API Compliance Checker|http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker] and think it would be a great addition to /dev-support. I propose that we use it to replace the JDiff wrappers we currently have there (since it does what JDiff does and more), and look into putting up automation at builds.apache.org to run the tool regularly (e.g. latest release of a particular branch vs. latest commit of that same branch). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11983) HRegion constructors should not create HLog
[
https://issues.apache.org/jira/browse/HBASE-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-11983:
-
Attachment: HBASE-11983.01.patch
Rebased to trunk, fixed some failing tests. See what BuildBot has to say.
HRegion constructors should not create HLog
Key: HBASE-11983
URL: https://issues.apache.org/jira/browse/HBASE-11983
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Enis Soztutar
Assignee: Nick Dimiduk
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-11983.00.patch, HBASE-11983.01.patch
We should get rid of HRegion creating its own HLog. It should ALWAYS get the
log from outside.
I think this was added for unit tests, but we should refrain from such
practice in the future (adding UT constructors always leads to weird and
critical bugs down the road). See recent: HBASE-11982, HBASE-11654.
Get rid of weird things like ignoreHLog:
{code}
/**
* @param ignoreHLog - true to skip generate new hlog if it is null, mostly
for createTable
*/
public static HRegion createHRegion(final HRegionInfo info, final Path
rootDir,
final Configuration conf,
final HTableDescriptor hTableDescriptor,
final HLog hlog,
final boolean initialize, final boolean
ignoreHLog)
{code}
We can unify all the createXX and newXX methods and separate creating a
region in the file system vs opening a region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11983) HRegion constructors should not create HLog
[
https://issues.apache.org/jira/browse/HBASE-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265256#comment-14265256
]
Nick Dimiduk commented on HBASE-11983:
--
[~enis] RB has been updated and I'm unable to create or attach files at present.
HRegion constructors should not create HLog
Key: HBASE-11983
URL: https://issues.apache.org/jira/browse/HBASE-11983
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Enis Soztutar
Assignee: Nick Dimiduk
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-11983.00.patch, HBASE-11983.01.patch
We should get rid of HRegion creating its own HLog. It should ALWAYS get the
log from outside.
I think this was added for unit tests, but we should refrain from such
practice in the future (adding UT constructors always leads to weird and
critical bugs down the road). See recent: HBASE-11982, HBASE-11654.
Get rid of weird things like ignoreHLog:
{code}
/**
* @param ignoreHLog - true to skip generate new hlog if it is null, mostly
for createTable
*/
public static HRegion createHRegion(final HRegionInfo info, final Path
rootDir,
final Configuration conf,
final HTableDescriptor hTableDescriptor,
final HLog hlog,
final boolean initialize, final boolean
ignoreHLog)
{code}
We can unify all the createXX and newXX methods and separate creating a
region in the file system vs opening a region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265279#comment-14265279
]
Ted Yu commented on HBASE-12728:
For AsyncPutter, please consider adding:
{code}
public synchronized add(ListPut puts);
{code}
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
Assignee: Solomon Duskis
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-8410) Basic quota support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265299#comment-14265299
]
Hadoop QA commented on HBASE-8410:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690142/HBASE-8410-master.4.patch
against master branch at commit 24bcebdeeb5c8025a1171fd73b5f38e666ae819b.
ATTACHMENT ID: 12690142
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2084 checkstyle errors (more than the master's current 2081 errors).
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRollPeriod
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.regionserver.TestJoinedScanners
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
org.apache.hadoop.hbase.TestHBaseTestingUtility
org.apache.hadoop.hbase.regionserver.TestRegionServerOnlineConfigChange
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.regionserver.TestRegionServerNoMaster
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay
org.apache.hadoop.hbase.regionserver.TestSCVFWithMiniCluster
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
org.apache.hadoop.hbase.regionserver.TestEncryptionRandomKeying
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush
{color:red}-1 core zombie tests{color}. There are 15 zombie test(s):
at
org.apache.hadoop.hbase.trace.TestHTraceHooks.testTraceCreateTable(TestHTraceHooks.java:71)
at
org.apache.hadoop.hbase.TestInfoServers.testMasterServerReadOnly(TestInfoServers.java:104)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort.testRSAbortWithUnflushedEdits(TestLogRollAbort.java:139)
at
org.apache.hadoop.hbase.regionserver.TestHRegion.testgetHDFSBlocksDistribution(TestHRegion.java:4146)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testCompactionRecordDoesntBlockRolling(TestLogRolling.java:574)
at
org.apache.hadoop.hbase.regionserver.TestRegionServerMetrics.testMutationsWithoutWal(TestRegionServerMetrics.java:192)
at
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad.testAtomicBulkLoad(TestHRegionServerBulkLoad.java:292)
at
org.apache.hadoop.hbase.regionserver.TestCompactionState.testMajorCompactionOnFamily(TestCompactionState.java:74)
at
org.apache.hadoop.hbase.regionserver.TestEncryptionKeyRotation.testMasterKeyRotation(TestEncryptionKeyRotation.java:162)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testRSSplitDaughtersAreOnlinedAfterShutdownHandling(TestSplitTransactionOnCluster.java:291)
at
org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload.testBulkLoad(TestScannerWithBulkload.java:76)
at
org.apache.hadoop.hbase.regionserver.TestTags.testTags(TestTags.java:120)
at
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster.testMergeWithReplicas(TestRegionMergeTransactionOnCluster.java:329)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12306//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12306//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
[jira] [Updated] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
[ https://issues.apache.org/jira/browse/HBASE-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12793: --- Fix Version/s: 1.1.0 Hadoop Flags: Reviewed Integrated to branch-1 and master branches. Esteban: Can you prepare patch for 0.98 ? [~enis]: Do you want this in 1.0.0 ? [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires - Key: HBASE-12793 URL: https://issues.apache.org/jira/browse/HBASE-12793 Project: HBase Issue Type: Sub-task Components: hbck Affects Versions: 1.0.0, 0.98.9 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0 Attachments: 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch This is subtask on HBASE-12131 in order to handle gracefully network partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11983) HRegion constructors should not create HLog
[
https://issues.apache.org/jira/browse/HBASE-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-11983:
-
Attachment: HBASE-11983.02.patch
Includes fixes for
- TestOfflineMetaRebuildBase
- TestReversibleScanners
- TestAtomicOperation
Notably, this delta touches the offline meta repair functionality in HBCK.
HRegion constructors should not create HLog
Key: HBASE-11983
URL: https://issues.apache.org/jira/browse/HBASE-11983
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Enis Soztutar
Assignee: Nick Dimiduk
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-11983.00.patch, HBASE-11983.01.patch,
HBASE-11983.02.patch
We should get rid of HRegion creating its own HLog. It should ALWAYS get the
log from outside.
I think this was added for unit tests, but we should refrain from such
practice in the future (adding UT constructors always leads to weird and
critical bugs down the road). See recent: HBASE-11982, HBASE-11654.
Get rid of weird things like ignoreHLog:
{code}
/**
* @param ignoreHLog - true to skip generate new hlog if it is null, mostly
for createTable
*/
public static HRegion createHRegion(final HRegionInfo info, final Path
rootDir,
final Configuration conf,
final HTableDescriptor hTableDescriptor,
final HLog hlog,
final boolean initialize, final boolean
ignoreHLog)
{code}
We can unify all the createXX and newXX methods and separate creating a
region in the file system vs opening a region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12799) ITAG fails with java.lang.RuntimeException if table does not exist
[
https://issues.apache.org/jira/browse/HBASE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265159#comment-14265159
]
Hudson commented on HBASE-12799:
SUCCESS: Integrated in HBase-1.1 #54 (See
[https://builds.apache.org/job/HBase-1.1/54/])
HBASE-12799 ITAG fails with java.lang.RuntimeException if table does not exist
(enis: rev e7795007b0b9bf270cd9df03c94a483104dd429b)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestAcidGuarantees.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
ITAG fails with java.lang.RuntimeException if table does not exist
--
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: hbase-12799_v1.patch
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1254)
at
org.apache.hadoop.hbase.TestAcidGuarantees.createTableIfMissing(TestAcidGuarantees.java:80)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:264)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.runTestFromCommandLine(IntegrationTestAcidGuarantees.java:58)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:112)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.main(IntegrationTestAcidGuarantees.java:115)
{code}
[~jmhsieh] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[ https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12339: Attachment: HBASE-12339.1.patch.txt Patch adds a log roller. relies on MockRegionServerServices, so we never report any flushes to the WAL. manually tested by watching the log of WALPerfEval for forced flushes and for size-based flushes. WAL performance evaluation tool doesn't roll logs - Key: HBASE-12339 URL: https://issues.apache.org/jira/browse/HBASE-12339 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.99.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12339.1.patch.txt the perf eval tool for the wal never sets up a log roller and instead used to just call the rollWriter method directly. Eventually it changed to call requestLogRoll instead of attempting to do the roll itself. requestLogRoll is the same method used internal to the wal code and it relies on there being a LogRoller to actually have rolls happen. (the method just notifies all of the listeners for hte wal that one of them should call the roll method.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[ https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12339: Status: Patch Available (was: In Progress) WAL performance evaluation tool doesn't roll logs - Key: HBASE-12339 URL: https://issues.apache.org/jira/browse/HBASE-12339 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.99.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12339.1.patch.txt the perf eval tool for the wal never sets up a log roller and instead used to just call the rollWriter method directly. Eventually it changed to call requestLogRoll instead of attempting to do the roll itself. requestLogRoll is the same method used internal to the wal code and it relies on there being a LogRoller to actually have rolls happen. (the method just notifies all of the listeners for hte wal that one of them should call the roll method.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12806: -- Fix Version/s: (was: 2.0.0) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12807) Implement 0.89fb-style roll on slow sync
Sean Busbey created HBASE-12807: --- Summary: Implement 0.89fb-style roll on slow sync Key: HBASE-12807 URL: https://issues.apache.org/jira/browse/HBASE-12807 Project: HBase Issue Type: Sub-task Components: wal Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0, 1.1.0 As a first step towards bringing down wal latency, implement the version of slow sync rolling done in the 0.89-fb branch. That is, instead of interrupting a slow sync operation, just request a log roll after whenever we detect one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[
https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265184#comment-14265184
]
stack commented on HBASE-12684:
---
The async rpc client is running slower, about 15% less throughput (see below
for more detail). Async client is doing await on future/promise or we are
allocating a direct buffer. You can't use a netty bytebuf pool? (Direct buffer
allocation is slow)
{code}
TestClient-3 #100 prio=5 os_prio=0 tid=0x7f2892193800 nid=0xe3d runnable
[0x7f286ad37000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
at
io.netty.buffer.UnpooledUnsafeDirectByteBuf.init(UnpooledUnsafeDirectByteBuf.java:69)
at
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)
at
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.writeRequest(AsyncRpcChannel.java:427)
at
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.callMethod(AsyncRpcChannel.java:323)
at
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.callMethodWithPromise(AsyncRpcChannel.java:345)
at
org.apache.hadoop.hbase.ipc.AsyncRpcClient.call(AsyncRpcClient.java:155)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:30860)
at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:873)
at org.apache.hadoop.hbase.client.HTable$4.call(HTable.java:864)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:881)
at
org.apache.hadoop.hbase.PerformanceEvaluation$RandomReadTest.testRow(PerformanceEvaluation.java:1253)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.testTimed(PerformanceEvaluation.java:1039)
at
org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:1021)
at
org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:1515)
at
org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:408)
at
org.apache.hadoop.hbase.PerformanceEvaluation$1.call(PerformanceEvaluation.java:403)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
I wrote 100M rows of various cell sizes between 0 and 8k bytes. This made for
about 30 regions. I hosted all 30 on one server and then on another I ran 5
clients in the one process as follows:
export HADOOP_CLASSPATH=$HOME/conf_hbase:`/home/stack/hbase/bin/hbase
classpath`
perf stat ${HOME}/hadoop/bin/hadoop --config ${HOME}/conf_hadoop
org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=100
randomRead 5 /tmp/$1.log
Nearly all requests are served from memory. Async client may do slightly less
gc.
Here are summary perf stats for the two client runs... they are close enough:
NOT-async:
{code}
695310.062208 task-clock#1.487 CPUs utilized
15,534,081 context-switches #0.022 M/sec
810,178 CPU-migrations#0.001 M/sec
84,042 page-faults #0.121 K/sec
1,092,725,117,827 cycles#1.572 GHz
[83.35%]
884,126,540,362 stalled-cycles-frontend # 80.91% frontend cycles idle
[83.30%]
620,320,998,946 stalled-cycles-backend# 56.77% backend cycles idle
[66.75%]
375,414,284,865 instructions #0.34 insns per cycle
#2.36 stalled cycles per insn
[83.43%]
70,798,813,916 branches # 101.823 M/sec
[83.29%]
3,979,009,879 branch-misses #5.62% of all branches
[83.31%]
467.591170612 seconds time elapsed
{code}
ASYNC
{code}
820257.269023 task-clock#1.496 CPUs utilized
15,698,021 context-switches
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[ https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265215#comment-14265215 ] Ted Yu commented on HBASE-12802: +1 Remove unnecessary Table.flushCommits() --- Key: HBASE-12802 URL: https://issues.apache.org/jira/browse/HBASE-12802 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12802.patch, HBASE-12802B.patch It looks like there are a lot of places where flushCommits() are called on tables that do not have autoFlush turned on. That might be a legacy concern from pre-autoFlush days. HBASE-12728 will likely result in removing the flushCommits() method from Table. The patch for this issue should remove all unnecessary calls to flushCommit() to prepare for the full fix.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[ https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265226#comment-14265226 ] stack commented on HBASE-12802: --- [~sduskis] That helps. Stick a version of your last comment as release note? After pasting the above, I'd say we don't need to do a bunch of runs... just get a clean hadoopqa before commit. Remove unnecessary Table.flushCommits() --- Key: HBASE-12802 URL: https://issues.apache.org/jira/browse/HBASE-12802 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12802.patch, HBASE-12802B.patch It looks like there are a lot of places where flushCommits() are called on tables that do not have autoFlush turned on. That might be a legacy concern from pre-autoFlush days. HBASE-12728 will likely result in removing the flushCommits() method from Table. The patch for this issue should remove all unnecessary calls to flushCommit() to prepare for the full fix.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8329) Limit compaction speed
[ https://issues.apache.org/jira/browse/HBASE-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265227#comment-14265227 ] Lars Hofhansl commented on HBASE-8329: -- Yep, that's the default (1 large and 1 small compaction thread). I can see a few small compaction threads, but it would depend heavily on the usage. Limit compaction speed -- Key: HBASE-8329 URL: https://issues.apache.org/jira/browse/HBASE-8329 Project: HBase Issue Type: Improvement Components: Compaction Reporter: binlijin Assignee: zhangduo Fix For: 2.0.0, 1.1.0 Attachments: HBASE-8329-10.patch, HBASE-8329-11.patch, HBASE-8329-12.patch, HBASE-8329-2-trunk.patch, HBASE-8329-3-trunk.patch, HBASE-8329-4-trunk.patch, HBASE-8329-5-trunk.patch, HBASE-8329-6-trunk.patch, HBASE-8329-7-trunk.patch, HBASE-8329-8-trunk.patch, HBASE-8329-9-trunk.patch, HBASE-8329-trunk.patch There is no speed or resource limit for compaction，I think we should add this feature especially when request burst. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Solomon Duskis updated HBASE-12802:
---
Description:
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are not
affected by autoFlush() and are not directly impacted by flushCommits() since
deletes are not added to the writeAysncBuffer.
was:It looks like there are a lot of places where flushCommits() are called
on tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix..
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265273#comment-14265273
]
Solomon Duskis commented on HBASE-12728:
Issue HBASE-12728 seems to suggest that setAutoFlushTo(), flushCommits() and
etc. should be removed from Table. HBASE-12802 cleans up some currently
unnecessary calls to flushCommits() which is a precursor to a new buffering
solution.
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
Assignee: Solomon Duskis
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as frequent as they would have been with
autoflush=true, except the synchronous write is moved from the operation
itself to the table close call which immediately follows.
More concretely :
```
// Given these two helpers ...
private HTableInterface getAutoFlushTable(String tableName) throws
IOException {
// (autoflush is true by default)
return storedConnection.getTable(tableName, executorService);
}
private HTableInterface getBufferedTable(String tableName) throws IOException
{
HTableInterface table = getAutoFlushTable(tableName);
table.setAutoFlush(false);
return table;
}
// it's my contention that these two methods would behave almost identically,
// except the first will hit a synchronous flush during the put call,
and the second will
// flush during the (hidden) close call on table.
private void writeAutoFlushed(Put somePut) throws IOException {
try (HTableInterface table = getAutoFlushTable(tableName)) {
table.put(somePut); // will do synchronous flush
}
}
private void writeBuffered(Put somePut) throws IOException {
try (HTableInterface table = getBufferedTable(tableName)) {
table.put(somePut);
} // auto-close will trigger synchronous flush
}
```
For buffered writes to actually provide a performance benefit to users, one
of two things must happen:
- The writeBuffer itself shouldn't live, flush and die with the lifecycle of
it's HTableInstance. If the writeBuffer were managed elsewhere and had a long
lifespan, this could cease to be an issue. However, if the same writeBuffer
is appended to by multiple tables, then some additional concurrency control
will be needed around it.
- Alternatively, there should be some pattern for having long-lived HTable
instances. However, since HTable is not thread-safe, we'd need multiple
instances, and a mechanism for leasing them out safely -- which sure sounds a
lot like the old HTablePool to me.
See discussion on mailing list here :
http://mail-archives.apache.org/mod_mbox/hbase-user/201412.mbox/%3CCAPdJLkEzmUQZ_kvD%3D8mrxi4V%3DhCmUp3g9MUZsddD%2Bmon%2BAvNtg%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
[
https://issues.apache.org/jira/browse/HBASE-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265286#comment-14265286
]
Hadoop QA commented on HBASE-12793:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690140/0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch
against master branch at commit 24bcebdeeb5c8025a1171fd73b5f38e666ae819b.
ATTACHMENT ID: 12690140
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
2082 checkstyle errors (more than the master's current 2081 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12308//console
This message is automatically generated.
[hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry
until hbase.hbck.close.timeout expires
-
Key: HBASE-12793
URL: https://issues.apache.org/jira/browse/HBASE-12793
Project: HBase
Issue Type: Sub-task
Components: hbck
Affects Versions: 1.0.0, 0.98.9
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Priority: Minor
Fix For: 2.0.0, 0.98.10, 1.0.1
Attachments:
0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch,
0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch,
0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch
This is subtask on HBASE-12131 in order to handle gracefully network
partitions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12806: -- Attachment: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
Esteban Gutierrez created HBASE-12806: - Summary: [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[ https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265223#comment-14265223 ] Ted Yu commented on HBASE-12339: +1 WAL performance evaluation tool doesn't roll logs - Key: HBASE-12339 URL: https://issues.apache.org/jira/browse/HBASE-12339 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.99.0 Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 2.0.0, 1.1.0 Attachments: HBASE-12339.1.patch.txt the perf eval tool for the wal never sets up a log roller and instead used to just call the rollWriter method directly. Eventually it changed to call requestLogRoll instead of attempting to do the roll itself. requestLogRoll is the same method used internal to the wal code and it relies on there being a LogRoller to actually have rolls happen. (the method just notifies all of the listeners for hte wal that one of them should call the roll method.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
[ https://issues.apache.org/jira/browse/HBASE-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-12793: -- Attachment: 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l-v0.98.patch Attached patch for 0.98, cc: [~apurtell] [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires - Key: HBASE-12793 URL: https://issues.apache.org/jira/browse/HBASE-12793 Project: HBase Issue Type: Sub-task Components: hbck Affects Versions: 1.0.0, 0.98.9 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0 Attachments: 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l-v0.98.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch This is subtask on HBASE-12131 in order to handle gracefully network partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12745) Visibility Labels: support visibility labels for user groups.
[
https://issues.apache.org/jira/browse/HBASE-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265361#comment-14265361
]
Hadoop QA commented on HBASE-12745:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690156/HBASE-12745-master-v3.patch
against master branch at commit 24bcebdeeb5c8025a1171fd73b5f38e666ae819b.
ATTACHMENT ID: 12690156
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12312//console
This message is automatically generated.
Visibility Labels: support visibility labels for user groups.
--
Key: HBASE-12745
URL: https://issues.apache.org/jira/browse/HBASE-12745
Project: HBase
Issue Type: Improvement
Components: security
Affects Versions: 1.0.0, 0.98.9, 0.99.2
Reporter: Jerry He
Assignee: Jerry He
Fix For: 2.0.0
Attachments: HBASE-12745-master-v1.patch,
HBASE-12745-master-v2.patch, HBASE-12745-master-v3.patch
The thinking is that we should support visibility labels to be associated
with user groups.
We will then be able grant visibility labels to a group in addition to
individual users, which provides convenience and usability.
We will use '@group' to denote a group name, as similarly done in
AcccessController.
For example,
{code}
set_auths '@group1', ['SECRET','PRIVATE']
{code}
{code}
get_auth '@group1'
{code}
A user belonging to 'group1' will have all the visibility labels granted to
'group1'
We'll also support super user groups as specified in hbase-site.xml.
The code update will mainly be on the server side VisibilityLabelService
implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12802:
---
Status: Open (was: Patch Available)
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265384#comment-14265384 ] Dima Spivak commented on HBASE-12806: - +1 from me. Good find, [~esteban]. [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck -- Key: HBASE-12806 URL: https://issues.apache.org/jira/browse/HBASE-12806 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 1.0.0, 2.0.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Attachments: 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch, 0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch TestHBaseFsck should wait until all regions have been assigned after the table has been created. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12802:
---
Fix Version/s: 1.0.0
Integrated to branch-1.0 as well.
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12658: --- Attachment: HBASE-12658-0.98.patch As promised. Should be no concerns now. Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 Key: HBASE-12658 URL: https://issues.apache.org/jira/browse/HBASE-12658 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.10 Attachments: HBASE-12658-0.98.patch, HBASE-12658-0.98.patch Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12791) HBase does not attempt to clean up an aborted split when the regionserver shutting down
[
https://issues.apache.org/jira/browse/HBASE-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265461#comment-14265461
]
Enis Soztutar commented on HBASE-12791:
---
Thanks Rajesh for working on this. A couple of comments for the patch:
- It is the right approach for the master to clean up after SplitTransaction
in case it fails to do so. I think this is good design.
- Can we move the cleaning logic away from RegionStates though (ideally to
SSH, or to a utility method).
- Can you add logging here:
{code}
+ } catch (IOException e) {
+// Nothing to do here.
+ }
{code}
- The hbck change seems costly. We already have all the regions from hdfs and
meta at that point no?
HBase does not attempt to clean up an aborted split when the regionserver
shutting down
---
Key: HBASE-12791
URL: https://issues.apache.org/jira/browse/HBASE-12791
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.0
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.0.1
Attachments: HBASE-12791.patch, HBASE-12791_v2.patch
HBase not cleaning the daughter region directories from HDFS if region
server shut down after creating the daughter region directories during the
split.
Here the logs.
- RS shutdown after creating the daughter regions.
{code}
2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385]
zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to
RS_ZK_REQUEST_REGION_SPLIT
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closing
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.: disabling compactions
flushes
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Updates disabled for region
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:41,516 INFO
[StoreCloserThread-t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.-1]
regionserver.HStore: Closed f
2014-12-31 09:05:41,518 INFO [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closed
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t dd9731ee43b104da565257ca1539aa8c
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,,1419996941401.dd9731ee43b104da565257ca1539aa8c.
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t 2e40a44511c0e187d357d651f13a1dab
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,row2,1419996941401.2e40a44511c0e187d357d651f13a1dab.
Wed Dec 31 09:06:30 IST 2014 Terminating regionserver
2014-12-31 09:06:30,465 INFO [Thread-8] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@42d2282e
{code}
- Skipping rollback if RS stopped or stopping so we end up in dirty daughter
regions in HDFS.
{code}
2014-12-31 09:07:49,547 INFO [regionserver60020-splits-1419996941385]
regionserver.SplitRequest: Skip rollback/cleanup of failed split of
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f. because server is stopped
java.io.InterruptedIOException: Interrupted after 0 tries on 350
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:156)
{code}
Because of this hbck always showing inconsistencies.
{code}
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/2e40a44511c0e187d357d651f13a1dab,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/dd9731ee43b104da565257ca1539aa8c,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
{code}
If we try to repair then we end up in overlap regions in hbase:meta. and both
daughter regions and parent are online.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-12779: -- Fix Version/s: (was: 0.94.27) SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265485#comment-14265485 ] Andrew Purtell commented on HBASE-12658: Thanks. Let me give [~enis] some time to comment on the tail of HBASE-12574 before commit to 0.98. Will commit tomorrow otherwise. Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 Key: HBASE-12658 URL: https://issues.apache.org/jira/browse/HBASE-12658 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.10 Attachments: HBASE-12658-0.98.patch, HBASE-12658-0.98.patch Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-12779. --- Resolution: Fixed And... 0.98 SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265490#comment-14265490 ] Andrew Purtell commented on HBASE-12779: Not 0.98? SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265501#comment-14265501 ] Lars Hofhansl commented on HBASE-12779: --- Cross commenting :) Just did 1.0, 1.1, 2.0 via cherry-picking and 0.98 with its own patch. SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12071: --- Status: In Progress (was: Patch Available) Separate out thread pool for Master - RegionServer communication -- Key: HBASE-12071 URL: https://issues.apache.org/jira/browse/HBASE-12071 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Sudarshan Kadambi Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch, HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch Over in HBASE-12028, there is a discussion about the case of a RegionServer still being alive despite all its handler threads being dead. One outcome of this is that the Master is left hanging on the RS for completion of various operations - such as region un-assignment when a table is disabled. Does it make sense to create a separate thread pool for communication between the Master and the RS? This addresses not just the case of the RPC handler threads terminating but also long-running queries or co-processor executions holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12071: --- Status: Patch Available (was: In Progress) Separate out thread pool for Master - RegionServer communication -- Key: HBASE-12071 URL: https://issues.apache.org/jira/browse/HBASE-12071 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Sudarshan Kadambi Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch, HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch Over in HBASE-12028, there is a discussion about the case of a RegionServer still being alive despite all its handler threads being dead. One outcome of this is that the Master is left hanging on the RS for completion of various operations - such as region un-assignment when a table is disabled. Does it make sense to create a separate thread pool for communication between the Master and the RS? This addresses not just the case of the RPC handler threads terminating but also long-running queries or co-processor executions holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12771) TestFailFast#testFastFail failing
[
https://issues.apache.org/jira/browse/HBASE-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manukranth Kolloju updated HBASE-12771:
---
Attachment: HBASE-12771-v2.patch
The patch was made against master, trying again.
TestFailFast#testFastFail failing
-
Key: HBASE-12771
URL: https://issues.apache.org/jira/browse/HBASE-12771
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: Manukranth Kolloju
Fix For: 1.0.0
Attachments: HBASE-12771-v1.patch, HBASE-12771-v2.patch
Fails on our internal rig and from time to time on apache. Here is latest:
{code}
org.apache.hadoop.hbase.client.TestFastFail.testFastFail
Failing for the past 1 build (Since Failed#654 )
Took 7.1 sec.
Error Message
Only few thread should ideally be waiting for the dead regionserver to be
coming back. numBlockedWorkers:155 threads that retried : 10
Stacktrace
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:155 threads that
retried : 10
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:270)
{code}
Opening this issue so can start tracking the fails. Looking in log, nothing
obvious. Will be back to this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12810) Update to htrace-incubating
Nick Dimiduk created HBASE-12810: Summary: Update to htrace-incubating Key: HBASE-12810 URL: https://issues.apache.org/jira/browse/HBASE-12810 Project: HBase Issue Type: Bug Reporter: Nick Dimiduk Fix For: 1.0.0, 2.0.0 HTrace has been accepted into incubator and there's been a flurry of RC's. This mostly just a package rename release. If there's a release in time, let's get 1.0 imports moved over to using the release out of incubator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12684) Add new AsyncRpcClient
[ https://issues.apache.org/jira/browse/HBASE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265349#comment-14265349 ] stack commented on HBASE-12684: --- bq. The await is not something that can change with the current blocking usage and the sync and async client should both block in a comparable way. Yes. Makes sense. Was just listing for completeness. I'm setup to try stuff for you; just say. Add new AsyncRpcClient -- Key: HBASE-12684 URL: https://issues.apache.org/jira/browse/HBASE-12684 Project: HBase Issue Type: Improvement Components: Client Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12684-DEBUG2.patch, HBASE-12684-DEBUG3.patch, HBASE-12684-v1.patch, HBASE-12684-v10.patch, HBASE-12684-v11.patch, HBASE-12684-v12.patch, HBASE-12684-v13.patch, HBASE-12684-v14.patch, HBASE-12684-v15.patch, HBASE-12684-v16.patch, HBASE-12684-v17.patch, HBASE-12684-v17.patch, HBASE-12684-v18.patch, HBASE-12684-v19.1.patch, HBASE-12684-v19.patch, HBASE-12684-v19.patch, HBASE-12684-v2.patch, HBASE-12684-v3.patch, HBASE-12684-v4.patch, HBASE-12684-v5.patch, HBASE-12684-v6.patch, HBASE-12684-v7.patch, HBASE-12684-v8.patch, HBASE-12684-v9.patch, HBASE-12684.patch With the changes in HBASE-12597 it is possible to add new RpcClients. This issue is about adding a new Async RpcClient which would enable HBase to do non blocking protobuf service communication. Besides delivering a new AsyncRpcClient I would also like to ask the question what it would take to replace the current RpcClient? This would enable to simplify async code in some next issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265356#comment-14265356 ] Hudson commented on HBASE-12785: FAILURE: Integrated in HBase-1.1 #55 (See [https://builds.apache.org/job/HBase-1.1/55/]) HBASE-12785 Use FutureTask to timeout the attempt to get the lock for hbck (tedyu: rev f9873622b86e7d8a912dfa15065e9bf96e10bacf) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: 12785-001.patch, 12785-002.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12802:
---
Attachment: 12802-1.0.patch
Patch for branch-1
[~enis]:
Do you want this in 1.0.0 ?
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265441#comment-14265441 ] Elliott Clark commented on HBASE-12658: --- I don't think that it's too big of an issue. However if it really is we should alias one so that there's only one copy of the string. Thanks for the work on the backport. Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 Key: HBASE-12658 URL: https://issues.apache.org/jira/browse/HBASE-12658 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.10 Attachments: HBASE-12658-0.98.patch Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[
https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265453#comment-14265453
]
Nick Dimiduk commented on HBASE-12071:
--
{noformat}
---
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
+++
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
{noformat}
Do not make changes to any generated code. Instead, this comment should be
updated in hbase-protocol/src/main/protobuf/RPC.proto and the protobuf files
regenerated.
Separate out thread pool for Master - RegionServer communication
--
Key: HBASE-12071
URL: https://issues.apache.org/jira/browse/HBASE-12071
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Sudarshan Kadambi
Assignee: Stephen Yuan Jiang
Fix For: 2.0.0
Attachments: HBASE-12071.v2-master.patch,
HBASE-12071.v2-master.patch, HBASE-12071.v3-master.patch
Over in HBASE-12028, there is a discussion about the case of a RegionServer
still being alive despite all its handler threads being dead. One outcome of
this is that the Master is left hanging on the RS for completion of various
operations - such as region un-assignment when a table is disabled. Does it
make sense to create a separate thread pool for communication between the
Master and the RS? This addresses not just the case of the RPC handler
threads terminating but also long-running queries or co-processor executions
holding up master operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
[ https://issues.apache.org/jira/browse/HBASE-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265454#comment-14265454 ] Hudson commented on HBASE-12793: SUCCESS: Integrated in HBase-1.1 #56 (See [https://builds.apache.org/job/HBase-1.1/56/]) HBASE-12793 [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires (Esteban) (tedyu: rev 071c3026f0325fc75d46e56cb2c6b569d7a374c9) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires - Key: HBASE-12793 URL: https://issues.apache.org/jira/browse/HBASE-12793 Project: HBase Issue Type: Sub-task Components: hbck Affects Versions: 1.0.0, 0.98.9 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0 Attachments: 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l-v0.98.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch This is subtask on HBASE-12131 in order to handle gracefully network partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265471#comment-14265471
]
Nick Dimiduk commented on HBASE-12802:
--
Hang on fellas, are these semantics really identical? The implementation of
doPut will check the buffer size and call backgroundFlushCommits when the
threshold is exceeded. That means that for-loop in put(List) will be chunked
according to buffer size. Am I reading this correctly?
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12771) TestFailFast#testFastFail failing
[
https://issues.apache.org/jira/browse/HBASE-12771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265481#comment-14265481
]
Manukranth Kolloju commented on HBASE-12771:
While testing for FastFail, I was randomly picking and killing the META server
and didn't account for the extra latencies that it would add. This was causing
the test to fail in branch 1. Seems like the region map relearning when META
goes down is faster in trunk than in branch 1. As a result, the test was never
failing in trunk but somehow failing in branch 1. Anyone knows why killing META
server is adding about an extra PAUSE_TIME worth delay in
RegionServerCallable.prepare ?
TestFailFast#testFastFail failing
-
Key: HBASE-12771
URL: https://issues.apache.org/jira/browse/HBASE-12771
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 1.0.0
Reporter: stack
Assignee: Manukranth Kolloju
Fails on our internal rig and from time to time on apache. Here is latest:
{code}
org.apache.hadoop.hbase.client.TestFastFail.testFastFail
Failing for the past 1 build (Since Failed#654 )
Took 7.1 sec.
Error Message
Only few thread should ideally be waiting for the dead regionserver to be
coming back. numBlockedWorkers:155 threads that retried : 10
Stacktrace
java.lang.AssertionError: Only few thread should ideally be waiting for the
dead regionserver to be coming back. numBlockedWorkers:155 threads that
retried : 10
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.client.TestFastFail.testFastFail(TestFastFail.java:270)
{code}
Opening this issue so can start tracking the fails. Looking in log, nothing
obvious. Will be back to this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-12779: - Assignee: Lars Hofhansl SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12779) SplitTransaction: Add metrics
[ https://issues.apache.org/jira/browse/HBASE-12779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-12779: -- Fix Version/s: (was: 1.0.1) 1.1.0 1.0.0 SplitTransaction: Add metrics - Key: HBASE-12779 URL: https://issues.apache.org/jira/browse/HBASE-12779 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0, 0.94.27 Attachments: 12779-v2-0.98.txt, 12779-v2.txt, 12779.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12574) Update replication metrics to not do so many map look ups.
[ https://issues.apache.org/jira/browse/HBASE-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265477#comment-14265477 ] Andrew Purtell commented on HBASE-12574: We could do branch-1.0 of course. But upgrade from 0.98 to something in 1.1 doesn't present a discontinuity against branch-1. Update replication metrics to not do so many map look ups. -- Key: HBASE-12574 URL: https://issues.apache.org/jira/browse/HBASE-12574 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.98.6.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.0.0, 2.0.0 Attachments: HBASE-12574.patch Replication is the last metrics area that still does a significant amount of hash map lookups. Lets re-factor that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265493#comment-14265493
]
Hudson commented on HBASE-12802:
FAILURE: Integrated in HBase-TRUNK #5995 (See
[https://builds.apache.org/job/HBase-TRUNK/5995/])
HBASE-12802 Remove unnecessary Table.flushCommits() (Solomon Duskis) (tedyu:
rev 2452d3861fb56c9ce7d748c32230b2df7798efc0)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestOpenTableInCoprocessor.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotCloneIndependence.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRebuildTestCore.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactionState.java
* hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannerResource.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFiltering.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java
*
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestScannersWithLabels.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
* hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabels.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFastFail.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestPrefixTree.java
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12793) [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires
[ https://issues.apache.org/jira/browse/HBASE-12793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265492#comment-14265492 ] Hudson commented on HBASE-12793: FAILURE: Integrated in HBase-TRUNK #5995 (See [https://builds.apache.org/job/HBase-TRUNK/5995/]) HBASE-12793 [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires (Esteban) (tedyu: rev 5b53a187d26ac54b7cd2c3d4a011bcc02931de99) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java [hbck] closeRegionSilentlyAndWait() should log cause of IOException and retry until hbase.hbck.close.timeout expires - Key: HBASE-12793 URL: https://issues.apache.org/jira/browse/HBASE-12793 Project: HBase Issue Type: Sub-task Components: hbck Affects Versions: 1.0.0, 0.98.9 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Minor Fix For: 2.0.0, 0.98.10, 1.0.1, 1.1.0 Attachments: 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0001-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l-v0.98.patch, 0002-HBASE-12793-hbck-closeRegionSilentlyAndWait-should-l.patch This is subtask on HBASE-12131 in order to handle gracefully network partitions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265518#comment-14265518
]
stack commented on HBASE-12802:
---
Where is the semantic changed [~ndimiduk]? This patch just changes tests
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12071: --- Attachment: HBASE-12071.v4-master.patch V4: address Nick's comments on generated file Separate out thread pool for Master - RegionServer communication -- Key: HBASE-12071 URL: https://issues.apache.org/jira/browse/HBASE-12071 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Sudarshan Kadambi Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch, HBASE-12071.v3-master.patch, HBASE-12071.v4-master.patch Over in HBASE-12028, there is a discussion about the case of a RegionServer still being alive despite all its handler threads being dead. One outcome of this is that the Master is left hanging on the RS for completion of various operations - such as region un-assignment when a table is disabled. Does it make sense to create a separate thread pool for communication between the Master and the RS? This addresses not just the case of the RPC handler threads terminating but also long-running queries or co-processor executions holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265353#comment-14265353
]
Hadoop QA commented on HBASE-12802:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690154/HBASE-12802B.patch
against master branch at commit 24bcebdeeb5c8025a1171fd73b5f38e666ae819b.
ATTACHMENT ID: 12690154
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 76 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12311//console
This message is automatically generated.
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
[jira] [Commented] (HBASE-12339) WAL performance evaluation tool doesn't roll logs
[
https://issues.apache.org/jira/browse/HBASE-12339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265352#comment-14265352
]
Hadoop QA commented on HBASE-12339:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690152/HBASE-12339.1.patch.txt
against master branch at commit 24bcebdeeb5c8025a1171fd73b5f38e666ae819b.
ATTACHMENT ID: 12690152
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12310//console
This message is automatically generated.
WAL performance evaluation tool doesn't roll logs
-
Key: HBASE-12339
URL: https://issues.apache.org/jira/browse/HBASE-12339
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 0.99.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-12339.1.patch.txt
the perf eval tool for the wal never sets up a log roller and instead used to
just call the rollWriter method directly.
Eventually it changed to call requestLogRoll instead of attempting to do the
roll itself. requestLogRoll is the same method used internal to the wal code
and it relies on there being a LogRoller to actually have rolls happen. (the
method just notifies all of the listeners for hte wal that one of them should
call the roll method.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12802:
---
Fix Version/s: 1.1.0
2.0.0
Hadoop Flags: Reviewed
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 2.0.0, 1.1.0
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12802) Remove unnecessary Table.flushCommits()
[
https://issues.apache.org/jira/browse/HBASE-12802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-12802.
Resolution: Fixed
Remove unnecessary Table.flushCommits()
---
Key: HBASE-12802
URL: https://issues.apache.org/jira/browse/HBASE-12802
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 2.0.0, 1.1.0
Attachments: 12802-1.0.patch, HBASE-12802.patch, HBASE-12802B.patch
It looks like there are a lot of places where flushCommits() are called on
tables that do not have autoFlush turned on. That might be a legacy concern
from pre-autoFlush days. HBASE-12728 will likely result in removing the
flushCommits() method from Table. The patch for this issue should remove all
unnecessary calls to flushCommit() to prepare for the full fix.
flushCommits() are only necessary after a setAutoFlushTo(false) is called.
Here's the relevant code from HTable.java:
{code:title=HTable.java|borderStyle=solid}
@Override
public void put(final Put put)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
doPut(put);
if (autoFlush) {
flushCommits();
}
}
/**
* {@inheritDoc}
*/
@Override
public void put(final ListPut puts)
throws InterruptedIOException, RetriesExhaustedWithDetailsException {
for (Put put : puts) {
doPut(put);
}
if (autoFlush) {
flushCommits();
}
}
{code}
Puts have implicit flushCommits() calls when autoFlush is true. Deletes are
not affected by autoFlush() and are not directly impacted by flushCommits()
since deletes are not added to the writeAysncBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12785) Use FutureTask to timeout the attempt to get the lock for hbck
[ https://issues.apache.org/jira/browse/HBASE-12785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265391#comment-14265391 ] Hudson commented on HBASE-12785: FAILURE: Integrated in HBase-TRUNK #5994 (See [https://builds.apache.org/job/HBase-TRUNK/5994/]) HBASE-12785 Use FutureTask to timeout the attempt to get the lock for hbck (tedyu: rev 37e1bb61f473ecc05af8fb1c547ec566acf7c6fb) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Use FutureTask to timeout the attempt to get the lock for hbck -- Key: HBASE-12785 URL: https://issues.apache.org/jira/browse/HBASE-12785 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.0.0, 1.1.0 Attachments: 12785-001.patch, 12785-002.patch In reviewing HBASE-12607, Sean pointed out: It would be nice if we used a [FutureTask|http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/FutureTask.html] to timeout the attempt to get the lock rather than wait the whole period and then fail. This issue is to address Sean's review comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12763) Make it so there must be WALs for a server to be marked dead
[ https://issues.apache.org/jira/browse/HBASE-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265406#comment-14265406 ] Enis Soztutar commented on HBASE-12763: --- bq. Ok. Fixed. Applied to branch-1.0 Ok, thanks Stack. Make it so there must be WALs for a server to be marked dead Key: HBASE-12763 URL: https://issues.apache.org/jira/browse/HBASE-12763 Project: HBase Issue Type: Sub-task Components: wal Reporter: stack Assignee: stack Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: 12746-v2-master-and-098.patch The patch for this issue is a subset of the patch attached to the parent. The parent solves a 1.0.0-specific issue but part of the patch needs applying to 0.98 and to master to fix an issue where Master on startup would think it was joining a cluster rather than undergoing a fresh start just because it came across a directory named for a server that was once running (the patch checks if the dir has WALs and if none, does not think the server a dead server). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11983) HRegion constructors should not create HLog
[
https://issues.apache.org/jira/browse/HBASE-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265412#comment-14265412
]
Hadoop QA commented on HBASE-11983:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690164/HBASE-11983.01.patch
against master branch at commit 37e1bb61f473ecc05af8fb1c547ec566acf7c6fb.
ATTACHMENT ID: 12690164
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 142
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestAtomicOperation.testPutAndCheckAndPutInParallel(TestAtomicOperation.java:549)
at
org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleRegionScanner(TestReversibleScanners.java:317)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12313//console
This message is automatically generated.
HRegion constructors should not create HLog
Key: HBASE-11983
URL: https://issues.apache.org/jira/browse/HBASE-11983
Project: HBase
Issue Type: Bug
Components: wal
Reporter: Enis Soztutar
Assignee: Nick Dimiduk
Labels: beginner
Fix For: 2.0.0, 1.1.0
Attachments: HBASE-11983.00.patch, HBASE-11983.01.patch,
HBASE-11983.02.patch
We should get rid of HRegion creating its own HLog. It should ALWAYS get the
log from outside.
I think this was added for unit tests, but we should refrain from such
practice in the future (adding UT constructors always leads to weird and
critical bugs down the road). See recent: HBASE-11982, HBASE-11654.
Get rid of weird things like ignoreHLog:
{code}
/**
* @param ignoreHLog - true to skip generate new hlog if it is null, mostly
for createTable
*/
public
[jira] [Updated] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[
https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12658:
---
Attachment: HBASE-12658-0.98.patch
Straightforward backport.
All changes are to private interfaces except where HBASE-11992 (backported int
0.98.9) changed o.a.h.h.regionserver.MetricsSource to
LimitedPrivate(REPLICATION). The changes of interest to MetricsSource are the
removal of public static final Strings:
{code}
@@ -32,31 +32,13 @@ import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.REPLICATION)
public class MetricsSource {
- public static final String SOURCE_SIZE_OF_LOG_QUEUE =
source.sizeOfLogQueue;
- public static final String SOURCE_AGE_OF_LAST_SHIPPED_OP =
source.ageOfLastShippedOp;
- public static final String SOURCE_LOG_EDITS_READ = source.logEditsRead;
- public static final String SOURCE_LOG_EDITS_FILTERED =
source.logEditsFiltered;
- public static final String SOURCE_SHIPPED_BATCHES = source.shippedBatches;
- public static final String SOURCE_SHIPPED_KBS = source.shippedKBs;
- public static final String SOURCE_SHIPPED_OPS = source.shippedOps;
- public static final String SOURCE_LOG_READ_IN_BYTES =
source.logReadInBytes;
{code}
No method signatures are modified. I don't think this presents a problem but
would like to highlight it for discussion. We could leave the string constants
in place if so.
Backport HBASE-12574 (Update replication metrics to not do so many map look
ups) to 0.98
Key: HBASE-12658
URL: https://issues.apache.org/jira/browse/HBASE-12658
Project: HBase
Issue Type: Task
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.98.10
Attachments: HBASE-12658-0.98.patch
Backport HBASE-12574 (Update replication metrics to not do so many map look
ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat
issues there, and we need to add Hadoop 1 compat module support.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12658: --- Status: Patch Available (was: Open) Setting Patch Available but Jenkins of course won't be able to apply the 0.98 patch to a precommit build. Replication tests including new test TestMetricsReplicationSourceFactoryImpl passes on Hadoop 2 and Hadoop 1 builds. Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 Key: HBASE-12658 URL: https://issues.apache.org/jira/browse/HBASE-12658 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.10 Attachments: HBASE-12658-0.98.patch Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12658) Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265452#comment-14265452 ] Andrew Purtell commented on HBASE-12658: bq. However if it really is we should alias one so that there's only one copy of the string. Actually let me do that just in case. Back with a new patch shortly. Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98 Key: HBASE-12658 URL: https://issues.apache.org/jira/browse/HBASE-12658 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.10 Attachments: HBASE-12658-0.98.patch Backport HBASE-12574 (Update replication metrics to not do so many map look ups) to 0.98. Nice enhancement. It's a refactor so need to check for compat issues there, and we need to add Hadoop 1 compat module support. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265450#comment-14265450
]
Hadoop QA commented on HBASE-12806:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12690167/0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch
against master branch at commit 5b53a187d26ac54b7cd2c3d4a011bcc02931de99.
ATTACHMENT ID: 12690167
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12314//console
This message is automatically generated.
[hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
--
Key: HBASE-12806
URL: https://issues.apache.org/jira/browse/HBASE-12806
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 1.0.0, 2.0.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Priority: Minor
Attachments:
0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch,
0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch
TestHBaseFsck should wait until all regions have been assigned after the
table has been created.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
