[jira] [Commented] (HBASE-13261) Make precommit job set jenkins description to ticket number
[ https://issues.apache.org/jira/browse/HBASE-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365271#comment-14365271 ] Sean Busbey commented on HBASE-13261: - I'm leaving this assigned to me as a reminder of something I want to do while waiting for builds/tests. If someone else gets time first, feel free to re-assign. Make precommit job set jenkins description to ticket number --- Key: HBASE-13261 URL: https://issues.apache.org/jira/browse/HBASE-13261 Project: HBase Issue Type: Task Components: build Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Labels: jenkins The hadoop precommit job sets the description of each job to the ticket it's covering. Makes it way easier to scan for what job covered something if you have to go looking. We should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13241) Add tests for group level grants
[ https://issues.apache.org/jira/browse/HBASE-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365269#comment-14365269 ] Srikanth Srungarapu commented on HBASE-13241: - [~ashish singhi] Thanks for so much for taking care of adding coverage for groups, which is a missing piece in the security testing. I'm desperate to give +1 on this as you can see from our offline communication. Had a little chat with [~mbertozzi] on v4, and I'm afraid that I couldn't let this pass through. * Please avoid checking whether grants are possible and verification of the scans in one test. * Please avoid doing things like USER1_TESTGROUP_TABLE.runAs. As you can see from the existing testing infrastructure, we generally create an action and use it with verifyAllowed and verifyDenied. * In short, as already suggested, I'm looking for something similar to TestAccessController#testGrantRevoke for verifying whether groups belonging to proper groups can grant and TestAccessController#testPostGrantRevokeAtQualifierLevel for verifying whether scans work assuming grants were already involved. * But if you have something in mind about why we can't do this, please let us know. Add tests for group level grants Key: HBASE-13241 URL: https://issues.apache.org/jira/browse/HBASE-13241 Project: HBase Issue Type: Improvement Components: security, test Reporter: Sean Busbey Assignee: Ashish Singhi Priority: Critical Attachments: HBASE-13241-v1.patch, HBASE-13241-v2.patch, HBASE-13241-v3.patch, HBASE-13241-v4.patch, HBASE-13241.patch We need to have tests for group-level grants for various scopes. ref: HBASE-13239 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13147) Load actual META table descriptor, don't use statically defined one.
[ https://issues.apache.org/jira/browse/HBASE-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-13147: - Attachment: HBASE-13147.v4.patch retry Load actual META table descriptor, don't use statically defined one. Key: HBASE-13147 URL: https://issues.apache.org/jira/browse/HBASE-13147 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Attachments: HBASE-13147-branch-1.patch, HBASE-13147-branch-1.v2.patch, HBASE-13147.patch, HBASE-13147.v2.patch, HBASE-13147.v3.patch, HBASE-13147.v4.patch, HBASE-13147.v4.patch In HBASE-13087 stumbled on the fact, that region servers don't see actual meta descriptor, they use their own, statically compiled. Need to fix that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5238) Add a log4j category for all edits to META/ROOT
[ https://issues.apache.org/jira/browse/HBASE-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-5238: Status: Open (was: Patch Available) Add a log4j category for all edits to META/ROOT --- Key: HBASE-5238 URL: https://issues.apache.org/jira/browse/HBASE-5238 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Andrey Stepachev Priority: Minor Labels: beginner Attachments: HBASE-5238.patch, HBASE-5238.patch, HBASE-5238.v2.patch, meta2.log Occasionally we run into bugs that have corrected META and written some bad data to META/ROOT but it's difficult to understand the order in which things happened. One option is to dump the HLog contents from the servers that hosted META at that time, but then it's interspersed with all other data. It would be nice to add a Log4j Logger to which we log all edits being applied to META and ROOT in textual form at DEBUG level. Then it would be easier to do a cluster-wide log grep to see what happened when. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5238) Add a log4j category for all edits to META/ROOT
[ https://issues.apache.org/jira/browse/HBASE-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-5238: Status: Patch Available (was: Open) Add a log4j category for all edits to META/ROOT --- Key: HBASE-5238 URL: https://issues.apache.org/jira/browse/HBASE-5238 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Andrey Stepachev Priority: Minor Labels: beginner Attachments: HBASE-5238.patch, HBASE-5238.patch, HBASE-5238.v2.patch, meta2.log Occasionally we run into bugs that have corrected META and written some bad data to META/ROOT but it's difficult to understand the order in which things happened. One option is to dump the HLog contents from the servers that hosted META at that time, but then it's interspersed with all other data. It would be nice to add a Log4j Logger to which we log all edits being applied to META and ROOT in textual form at DEBUG level. Then it would be easier to do a cluster-wide log grep to see what happened when. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13229) Specify bash for local-regionservers.sh and local-master-backup.sh
[
https://issues.apache.org/jira/browse/HBASE-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13229:
Resolution: Fixed
Fix Version/s: 0.98.12
0.94.27
1.1.0
1.0.1
2.0.0
Status: Resolved (was: Patch Available)
Pushed, then realized I missed fixing the commit message to include the jira
reference.
Here's the set of commits:
[master|https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=commit;h=ca8876a9f2534f2ab0b416aecb4ac476da9747f8],
[branch-1|https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=commit;h=58ab201be341f02829286f036a7401d0806eb999],
[branch-1.0|https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=commit;h=6b37ae3d77e68458cae385b11163ac5108af7655],
[0.98|https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=commit;h=02a1f3a5ba46c4d8d72135607e5d20355d1061a2],
[0.94|https://git1-us-west.apache.org/repos/asf?p=hbase.git;a=commit;h=9e6b65226a59aa13d470b6d1e83eb7035ce9].
I filed HBASE-13263 to help make sure I don't do this again.
Specify bash for local-regionservers.sh and local-master-backup.sh
--
Key: HBASE-13229
URL: https://issues.apache.org/jira/browse/HBASE-13229
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.94.0, 0.98.0, 1.0.0
Reporter: Gustavo Anatoly
Assignee: Gustavo Anatoly
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.94.27, 0.98.12
Attachments: HBASE-12339-v1.patch, HBASE-13229-v2.patch,
HBASE-13229.patch
Running the following line, using /bin/sh:
$ bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/ start 1 2
3 4 5
Produces the output below:
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
Considering:
{code}
if [[ $i =~ ^[0-9]+$ ]]; then
run_master $cmd $i
else
echo Invalid argument
fi
{code}
The reasons is that the regex operator =~ doesn't have compatibility with
/bin/sh but works running /bin/bash
$ bash -x bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/
start 1 2 3 4 5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13258) Promote TestHRegion to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365353#comment-14365353 ] zhangduo commented on HBASE-13258: -- I follow [~busbey]' suggestion, copy a new jenkins job https://builds.apache.org/job/HBase-TRUNK-jacoco/1/console and debug there. Now I have to go to sleep, hope I can make some progress tomorrow. Promote TestHRegion to LargeTests - Key: HBASE-13258 URL: https://issues.apache.org/jira/browse/HBASE-13258 Project: HBase Issue Type: Sub-task Components: test Reporter: zhangduo Assignee: zhangduo Attachments: HBASE-13258-addendum.patch, HBASE-13258.patch, HBASE-13258.patch It always timeout we I tried to get a coverage report locally. The problem is testWritesWhileGetting, it runs extremely slow when jacoco agent enabled(not a bug, there is progress). Since it has a VerySlowRegionServerTests annotation on it, I think it is OK to promote it to LargeTests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13263) Client side git hooks to enforce commit message best practices
[ https://issues.apache.org/jira/browse/HBASE-13263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365350#comment-14365350 ] Sean Busbey commented on HBASE-13263: - start with the solution [cloudstack uses for their contributors|https://cwiki.apache.org/confluence/display/CLOUDSTACK/Git#Git-CommitMessages] Client side git hooks to enforce commit message best practices -- Key: HBASE-13263 URL: https://issues.apache.org/jira/browse/HBASE-13263 Project: HBase Issue Type: Task Components: build Reporter: Sean Busbey Assignee: Sean Busbey Add client side git hooks to enforce commit message best practices * jira listed * --signed-off for contributed patches -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13258) Promote TestHRegion to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365267#comment-14365267 ] Hudson commented on HBASE-13258: ABORTED: Integrated in HBase-TRUNK #6271 (See [https://builds.apache.org/job/HBase-TRUNK/6271/]) HBASE-13258 addendum add log and reduce testCount in TestHRegion.testWritesWhileGetting (zhangduo: rev 7b7b0bf9dd08950b314ea60cd2b2d24deff8cd15) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java Promote TestHRegion to LargeTests - Key: HBASE-13258 URL: https://issues.apache.org/jira/browse/HBASE-13258 Project: HBase Issue Type: Sub-task Components: test Reporter: zhangduo Assignee: zhangduo Attachments: HBASE-13258-addendum.patch, HBASE-13258.patch, HBASE-13258.patch It always timeout we I tried to get a coverage report locally. The problem is testWritesWhileGetting, it runs extremely slow when jacoco agent enabled(not a bug, there is progress). Since it has a VerySlowRegionServerTests annotation on it, I think it is OK to promote it to LargeTests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13261) Make precommit job set jenkins description to ticket number
Sean Busbey created HBASE-13261: --- Summary: Make precommit job set jenkins description to ticket number Key: HBASE-13261 URL: https://issues.apache.org/jira/browse/HBASE-13261 Project: HBase Issue Type: Task Components: build Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor The hadoop precommit job sets the description of each job to the ticket it's covering. Makes it way easier to scan for what job covered something if you have to go looking. We should do the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13176) Flakey TestZooKeeper test.
[
https://issues.apache.org/jira/browse/HBASE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365296#comment-14365296
]
Andrey Stepachev commented on HBASE-13176:
--
Done for branch-1 and master. Seems that 0.98 shouldn't have this problem due
of lack ZkSplitLogWorkerCoordination.
Flakey TestZooKeeper test.
--
Key: HBASE-13176
URL: https://issues.apache.org/jira/browse/HBASE-13176
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Attachments: HBASE-13176.patch
Test fails with wrong number of listeners:
{code}
Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 202.248 sec
FAILURE! - in org.apache.hadoop.hbase.TestZooKeeper
testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(org.apache.hadoop.hbase.TestZooKeeper)
Time elapsed: 48.687 sec FAILURE!
java.lang.AssertionError: expected:16 but was:15
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:518)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-5238) Add a log4j category for all edits to META/ROOT
[ https://issues.apache.org/jira/browse/HBASE-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365313#comment-14365313 ] Andrey Stepachev commented on HBASE-5238: - removed META: prefix. it is off by default, because it logs into DEBUG facility. So basically it can be turned ON whenever level is changed to DEBUG. Add a log4j category for all edits to META/ROOT --- Key: HBASE-5238 URL: https://issues.apache.org/jira/browse/HBASE-5238 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Andrey Stepachev Priority: Minor Labels: beginner Attachments: HBASE-5238.patch, HBASE-5238.patch, meta2.log Occasionally we run into bugs that have corrected META and written some bad data to META/ROOT but it's difficult to understand the order in which things happened. One option is to dump the HLog contents from the servers that hosted META at that time, but then it's interspersed with all other data. It would be nice to add a Log4j Logger to which we log all edits being applied to META and ROOT in textual form at DEBUG level. Then it would be easier to do a cluster-wide log grep to see what happened when. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13258) Promote TestHRegion to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365312#comment-14365312 ] stack commented on HBASE-13258: --- +1 on your experiments [~Apache9] Promote TestHRegion to LargeTests - Key: HBASE-13258 URL: https://issues.apache.org/jira/browse/HBASE-13258 Project: HBase Issue Type: Sub-task Components: test Reporter: zhangduo Assignee: zhangduo Attachments: HBASE-13258-addendum.patch, HBASE-13258.patch, HBASE-13258.patch It always timeout we I tried to get a coverage report locally. The problem is testWritesWhileGetting, it runs extremely slow when jacoco agent enabled(not a bug, there is progress). Since it has a VerySlowRegionServerTests annotation on it, I think it is OK to promote it to LargeTests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5238) Add a log4j category for all edits to META/ROOT
[ https://issues.apache.org/jira/browse/HBASE-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-5238: Attachment: HBASE-5238.v2.patch Add a log4j category for all edits to META/ROOT --- Key: HBASE-5238 URL: https://issues.apache.org/jira/browse/HBASE-5238 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Andrey Stepachev Priority: Minor Labels: beginner Attachments: HBASE-5238.patch, HBASE-5238.patch, HBASE-5238.v2.patch, meta2.log Occasionally we run into bugs that have corrected META and written some bad data to META/ROOT but it's difficult to understand the order in which things happened. One option is to dump the HLog contents from the servers that hosted META at that time, but then it's interspersed with all other data. It would be nice to add a Log4j Logger to which we log all edits being applied to META and ROOT in textual form at DEBUG level. Then it would be easier to do a cluster-wide log grep to see what happened when. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13176) Flakey TestZooKeeper test.
[
https://issues.apache.org/jira/browse/HBASE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Stepachev updated HBASE-13176:
-
Resolution: Fixed
Fix Version/s: 1.1.1
1.0.1
2.0.0
Status: Resolved (was: Patch Available)
Flakey TestZooKeeper test.
--
Key: HBASE-13176
URL: https://issues.apache.org/jira/browse/HBASE-13176
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Fix For: 2.0.0, 1.0.1, 1.1.1
Attachments: HBASE-13176.patch
Test fails with wrong number of listeners:
{code}
Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 202.248 sec
FAILURE! - in org.apache.hadoop.hbase.TestZooKeeper
testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(org.apache.hadoop.hbase.TestZooKeeper)
Time elapsed: 48.687 sec FAILURE!
java.lang.AssertionError: expected:16 but was:15
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:518)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13263) Client side git hooks to enforce commit message best practices
Sean Busbey created HBASE-13263: --- Summary: Client side git hooks to enforce commit message best practices Key: HBASE-13263 URL: https://issues.apache.org/jira/browse/HBASE-13263 Project: HBase Issue Type: Task Components: build Reporter: Sean Busbey Assignee: Sean Busbey Add client side git hooks to enforce commit message best practices * jira listed * --signed-off for contributed patches -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366639#comment-14366639 ] zhangduo commented on HBASE-13257: -- https://builds.apache.org/job/HBase-TRUNK-jacoco/ Findbugs and checkstyle result are also shown. There are some high priority findbugs warnings. In my experience, a high priority findbugs warning is usually a bug or a very bad practice. Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13269) Remove result array preallocation to avoid OOME with large scan caching values
[ https://issues.apache.org/jira/browse/HBASE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366594#comment-14366594 ] stack commented on HBASE-13269: --- Does what JIRA promises so +1 (but pity we can't presize -- to do later when we have more 'sophisticated' means as per [~jonathan.lawlor]) Remove result array preallocation to avoid OOME with large scan caching values -- Key: HBASE-13269 URL: https://issues.apache.org/jira/browse/HBASE-13269 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.0.1, 0.98.12 Attachments: HBASE-13269-0.98.patch, HBASE-13269-1.0.patch Scan#setCaching(Integer.MAX_VALUE) will likely terminate the regionserver with an OOME due to preallocation of the result array according to this parameter. We should limit the preallocation to some sane value. Definitely affects 0.98 (fix needed to HRegionServer) and 1.0.x (fix needed to RsRPCServices), not sure about later versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366602#comment-14366602 ] zhangduo commented on HBASE-13257: -- If this slow down is acceptable then maybe just replace the HBase-TRUNK config? It is important to keep PreCommit run fast so we should not do this in PreCommit build, but for TRUNK I think it is OK? [~stack] Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[
https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366644#comment-14366644
]
Sean Busbey commented on HBASE-13257:
-
{quote}
It is important to keep PreCommit run fast so we should not do this in
PreCommit build, but for TRUNK I think it is OK?
{quote}
I was really hoping to get this on precommit. The PreCommit job is already
really slow (like 1-2 hours) and we already run findbugs during it.
Show coverage report on jenkins
---
Key: HBASE-13257
URL: https://issues.apache.org/jira/browse/HBASE-13257
Project: HBase
Issue Type: Task
Reporter: zhangduo
Assignee: zhangduo
Priority: Minor
Think of showing jacoco coverage report on https://builds.apache.org .
And there is an advantage of showing it on jenkins that the jenkins jacoco
plugin can handle cross module coverage.
Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still
pending.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366589#comment-14366589 ] stack commented on HBASE-13257: --- What you thinking [~Apache9]? Run this once a week? It looks very nice. Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13230) [mob] reads hang when trying to read rows with large mobs (10MB)
[ https://issues.apache.org/jira/browse/HBASE-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13230: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) thanks matteo [mob] reads hang when trying to read rows with large mobs (10MB) - Key: HBASE-13230 URL: https://issues.apache.org/jira/browse/HBASE-13230 Project: HBase Issue Type: Bug Components: mob Affects Versions: hbase-11339 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: hbase-11339 Attachments: HBASE-13230.hbase-11339.patch Using load tests tool to read and write out 5MB, 10MB, 20MB objects works fine, but problems are encountered when trying to read values 20MB. This is due to the default protobuf size limit of 64MB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13223) Add testMoveMeta to IntegrationTestMTTR
[ https://issues.apache.org/jira/browse/HBASE-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364739#comment-14364739 ] Hudson commented on HBASE-13223: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #858 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/858/]) HBASE-13223 Add testMoveMeta to IntegrationTestMTTR (apurtell: rev 2a2884f6bf6c682675fc7b733fdf637d1effa2f0) * hbase-it/src/test/java/org/apache/hadoop/hbase/mttr/IntegrationTestMTTR.java Add testMoveMeta to IntegrationTestMTTR --- Key: HBASE-13223 URL: https://issues.apache.org/jira/browse/HBASE-13223 Project: HBase Issue Type: Improvement Components: integration tests Reporter: Dima Spivak Assignee: Dima Spivak Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-13223_master_v1.patch It would be nice to add a new test case to IntegrationTestMTTR that would trigger a move of the regions of hbase:meta and evaluate the MTTR in the aftermath. Pretty straightforward addition to the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12908) Typos in MemStoreFlusher javadocs
[ https://issues.apache.org/jira/browse/HBASE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12908: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1. thanks for the patch edvin. Typos in MemStoreFlusher javadocs - Key: HBASE-12908 URL: https://issues.apache.org/jira/browse/HBASE-12908 Project: HBase Issue Type: Bug Components: documentation Reporter: Edvin Malinovskis Priority: Trivial Labels: javadoc, typo Attachments: HBASE-12908.patch The javadocs for the `flushRegion` methods say that _there will be accompanying log messages explaining why the *log* was not flushed._; when it should say _... why the *region* was not flushed._ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13235) Revisit the security auditing semantics.
[
https://issues.apache.org/jira/browse/HBASE-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364743#comment-14364743
]
Hadoop QA commented on HBASE-13235:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705011/HBASE-13235_v2.patch
against master branch at commit bf9e32d59d65afa6425f03e488959c8883038801.
ATTACHMENT ID: 12705011
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13280//console
This message is automatically generated.
Revisit the security auditing semantics.
Key: HBASE-13235
URL: https://issues.apache.org/jira/browse/HBASE-13235
Project: HBase
Issue Type: Improvement
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Attachments: HBASE-13235.patch, HBASE-13235_v2.patch,
HBASE-13235_v2.patch
More specifically, the following things need a closer look. (Will include
more based on feedback and/or suggestions)
* Table name (say test) instead of fully qualified table name(default:test)
being used.
* Right now, we're using the scope to be similar to arguments for operation.
Would be better to decouple the arguments for operation and scope involved in
checking. For e.g. say for createTable, we have the following audit log
{code}
Access denied for user esteban; reason: Insufficient permissions; remote
address: /10.20.30.1; request: createTable; context: (user=srikanth@XXX,
scope=default, action=CREATE)
{code}
The scope was rightly being used as default
[jira] [Commented] (HBASE-13258) Promote TestHRegion to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364741#comment-14364741 ] zhangduo commented on HBASE-13258: -- Pushed to master. Now let me try to modify jenkins config. Promote TestHRegion to LargeTests - Key: HBASE-13258 URL: https://issues.apache.org/jira/browse/HBASE-13258 Project: HBase Issue Type: Sub-task Components: test Reporter: zhangduo Assignee: zhangduo Attachments: HBASE-13258.patch, HBASE-13258.patch It always timeout we I tried to get a coverage report locally. The problem is testWritesWhileGetting, it runs extremely slow when jacoco agent enabled(not a bug, there is progress). Since it has a VerySlowRegionServerTests annotation on it, I think it is OK to promote it to LargeTests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364754#comment-14364754
]
Mikhail Antonov commented on HBASE-13103:
-
Some notes as I'm sketching the draft:
- generally looks like the basic balancing architecture would just work - a
chore on master, interface + pluggable initialization, main invocation gets
kicked in HMaster, period and cutoff time limits.
- runs on per-table basis (for first cut - could just do all or nothing, then
add normalization params to table level configuration if needed)
- normalizer computes list of normalization plans (which are simply, either
split R1, or merge R1 and R1), those plans then executed one by one, I guess we
don't want more than one merge or split going on the table in most cases?
Execution of plan is simply figuring currently assigned HRS and requesting
split over rpc.
- whole thing is stateless, if master crashed during normalization, on next
scheduled iteration it will be recomputed anyway
[~ndimiduk] thoughts?
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Brainstorming
Components: Usability
Reporter: Nick Dimiduk
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13258) Promote TestHRegion to LargeTests
[
https://issues.apache.org/jira/browse/HBASE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364695#comment-14364695
]
Hadoop QA commented on HBASE-13258:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704999/HBASE-13258.patch
against master branch at commit bf9e32d59d65afa6425f03e488959c8883038801.
ATTACHMENT ID: 12704999
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13279//console
This message is automatically generated.
Promote TestHRegion to LargeTests
-
Key: HBASE-13258
URL: https://issues.apache.org/jira/browse/HBASE-13258
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: zhangduo
Assignee: zhangduo
Attachments: HBASE-13258.patch, HBASE-13258.patch
It always timeout we I tried to get a coverage report locally. The problem is
testWritesWhileGetting, it runs extremely slow when jacoco agent enabled(not
a bug, there is progress).
Since it has a VerySlowRegionServerTests annotation on it, I think it is OK
to promote it to LargeTests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12908) Typos in MemStoreFlusher javadocs
[ https://issues.apache.org/jira/browse/HBASE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364727#comment-14364727 ] Jonathan Hsieh commented on HBASE-12908: I'm currently having problems adding edvin as a contributor, will try again later. Typos in MemStoreFlusher javadocs - Key: HBASE-12908 URL: https://issues.apache.org/jira/browse/HBASE-12908 Project: HBase Issue Type: Bug Components: documentation Reporter: Edvin Malinovskis Priority: Trivial Labels: javadoc, typo Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-12908.patch The javadocs for the `flushRegion` methods say that _there will be accompanying log messages explaining why the *log* was not flushed._; when it should say _... why the *region* was not flushed._ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12908) Typos in MemStoreFlusher javadocs
[ https://issues.apache.org/jira/browse/HBASE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12908: --- Fix Version/s: 0.98.12 1.1.0 1.0.1 2.0.0 Typos in MemStoreFlusher javadocs - Key: HBASE-12908 URL: https://issues.apache.org/jira/browse/HBASE-12908 Project: HBase Issue Type: Bug Components: documentation Reporter: Edvin Malinovskis Priority: Trivial Labels: javadoc, typo Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-12908.patch The javadocs for the `flushRegion` methods say that _there will be accompanying log messages explaining why the *log* was not flushed._; when it should say _... why the *region* was not flushed._ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13233) add hbase-11339 branch to the patch testing script
[ https://issues.apache.org/jira/browse/HBASE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13233: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) thanks sean. add hbase-11339 branch to the patch testing script -- Key: HBASE-13233 URL: https://issues.apache.org/jira/browse/HBASE-13233 Project: HBase Issue Type: Task Affects Versions: 2.0.0, hbase-11339 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 2.0.0 Attachments: hbase-13233.patch adding hbase-11339 to the BRANCH_NAMES so we can use the apache bot to test patches on that branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13230) [mob] reads hang when trying to read rows with large mobs (10MB)
[ https://issues.apache.org/jira/browse/HBASE-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13230: --- Status: Patch Available (was: In Progress) [mob] reads hang when trying to read rows with large mobs (10MB) - Key: HBASE-13230 URL: https://issues.apache.org/jira/browse/HBASE-13230 Project: HBase Issue Type: Bug Components: mob Affects Versions: hbase-11339 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: hbase-11339 Attachments: HBASE-13230.hbase-11339.patch Using load tests tool to read and write out 5MB, 10MB, 20MB objects works fine, but problems are encountered when trying to read values 20MB. This is due to the default protobuf size limit of 64MB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13233) add hbase-11339 branch to the patch testing script
[ https://issues.apache.org/jira/browse/HBASE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-13233: --- Priority: Minor (was: Major) add hbase-11339 branch to the patch testing script -- Key: HBASE-13233 URL: https://issues.apache.org/jira/browse/HBASE-13233 Project: HBase Issue Type: Task Affects Versions: 2.0.0, hbase-11339 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Priority: Minor Fix For: 2.0.0 Attachments: hbase-13233.patch adding hbase-11339 to the BRANCH_NAMES so we can use the apache bot to test patches on that branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HBASE-13230) [mob] reads hang when trying to read rows with large mobs (10MB)
[ https://issues.apache.org/jira/browse/HBASE-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-13230 started by Jonathan Hsieh. -- [mob] reads hang when trying to read rows with large mobs (10MB) - Key: HBASE-13230 URL: https://issues.apache.org/jira/browse/HBASE-13230 Project: HBase Issue Type: Bug Components: mob Affects Versions: hbase-11339 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: hbase-11339 Attachments: HBASE-13230.hbase-11339.patch Using load tests tool to read and write out 5MB, 10MB, 20MB objects works fine, but problems are encountered when trying to read values 20MB. This is due to the default protobuf size limit of 64MB. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364750#comment-14364750 ] Hudson commented on HBASE-11339: ABORTED: Integrated in HBase-TRUNK #6269 (See [https://builds.apache.org/job/HBase-TRUNK/6269/]) HBASE-13233 add hbase-11339 branch to the patch testing script (jmhsieh: rev e192f5ed39911d180287730315db51f18f0e5018) * dev-support/test-patch.properties HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user guide_v2.docx, MOB user guide_v3.docx, MOB user guide_v4.docx, MOB user guide_v5.docx, hbase-11339-in-dev.patch, merge-150212.patch, merge.150212b.patch, merge.150212c.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364749#comment-14364749 ] Enis Soztutar commented on HBASE-6721: -- This is the reason why we have started the discussion thread on dev@ some time ago: http://search-hadoop.com/m/DHED47vIer1. RS group based assignment is already a plugin implementation, but we thought that bringing it as a core feature made sense because it will take some time to have full QOS and isolation. In the mean time, RS group based assignments are a reasonable solution that solves a real problem (although at Yahoo!'s scale). However, we are seeing more interest in this patch from other users as well. If there is strong objection for this being a core feature, I think we should still commit this as a co-proc based implementation. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Attachments: 6721-master-webUI.patch, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721_10.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, HBASE-6721_94_7.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, HBASE-6721_trunk2.patch In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13233) add hbase-11339 branch to the patch testing script
[ https://issues.apache.org/jira/browse/HBASE-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364752#comment-14364752 ] Hudson commented on HBASE-13233: ABORTED: Integrated in HBase-TRUNK #6269 (See [https://builds.apache.org/job/HBase-TRUNK/6269/]) HBASE-13233 add hbase-11339 branch to the patch testing script (jmhsieh: rev e192f5ed39911d180287730315db51f18f0e5018) * dev-support/test-patch.properties add hbase-11339 branch to the patch testing script -- Key: HBASE-13233 URL: https://issues.apache.org/jira/browse/HBASE-13233 Project: HBase Issue Type: Task Affects Versions: 2.0.0, hbase-11339 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Priority: Minor Fix For: 2.0.0 Attachments: hbase-13233.patch adding hbase-11339 to the BRANCH_NAMES so we can use the apache bot to test patches on that branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13258) Promote TestHRegion to LargeTests
[ https://issues.apache.org/jira/browse/HBASE-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364751#comment-14364751 ] Hudson commented on HBASE-13258: ABORTED: Integrated in HBase-TRUNK #6269 (See [https://builds.apache.org/job/HBase-TRUNK/6269/]) HBASE-13258 Promote TestHRegion to LargeTests (zhangduo: rev 2b08653a79a8ed0ae8501a110b79f9ea23e808d4) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java Promote TestHRegion to LargeTests - Key: HBASE-13258 URL: https://issues.apache.org/jira/browse/HBASE-13258 Project: HBase Issue Type: Sub-task Components: test Reporter: zhangduo Assignee: zhangduo Attachments: HBASE-13258.patch, HBASE-13258.patch It always timeout we I tried to get a coverage report locally. The problem is testWritesWhileGetting, it runs extremely slow when jacoco agent enabled(not a bug, there is progress). Since it has a VerySlowRegionServerTests annotation on it, I think it is OK to promote it to LargeTests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12908) Typos in MemStoreFlusher javadocs
[ https://issues.apache.org/jira/browse/HBASE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364753#comment-14364753 ] Hudson commented on HBASE-12908: ABORTED: Integrated in HBase-TRUNK #6269 (See [https://builds.apache.org/job/HBase-TRUNK/6269/]) HBASE-12908 Typos in MemstoreFlusher javadocs (Edvin Malinovskis) (jmhsieh: rev 65ad19ddf7848dcc811b7968fb31fe6fc05885d8) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java Typos in MemStoreFlusher javadocs - Key: HBASE-12908 URL: https://issues.apache.org/jira/browse/HBASE-12908 Project: HBase Issue Type: Bug Components: documentation Reporter: Edvin Malinovskis Priority: Trivial Labels: javadoc, typo Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-12908.patch The javadocs for the `flushRegion` methods say that _there will be accompanying log messages explaining why the *log* was not flushed._; when it should say _... why the *region* was not flushed._ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13212) Procedure V2 - master Create/Modify/Delete namespace
[ https://issues.apache.org/jira/browse/HBASE-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364649#comment-14364649 ] Stephen Yuan Jiang commented on HBASE-13212: Yeah, thanks for pointing it out - update the title and description. Procedure V2 - master Create/Modify/Delete namespace Key: HBASE-13212 URL: https://issues.apache.org/jira/browse/HBASE-13212 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Original Estimate: 168h Remaining Estimate: 168h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the create/modify/delete namespace handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13212) Procedure V2 - master Create/Modify/Delete namespace
[ https://issues.apache.org/jira/browse/HBASE-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-13212: --- Description: master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the create/modify/delete namespace handlers with the procedure version. was: master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the create/delete namespace handlers with the procedure version. Summary: Procedure V2 - master Create/Modify/Delete namespace (was: Procedure V2 - master Create/Delete namespace) Procedure V2 - master Create/Modify/Delete namespace Key: HBASE-13212 URL: https://issues.apache.org/jira/browse/HBASE-13212 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0, 1.1.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Original Estimate: 168h Remaining Estimate: 168h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the create/modify/delete namespace handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365848#comment-14365848 ] stack commented on HBASE-13259: --- [~zeocio] is it not configurable now? On the patch, nice and clean. Here where you do this... fileSize = roundUp(capacity, bufferSize); and then... raf.setLength(fileSize); ... any chance of us reading extra bytes off the end of the file? End of an hfile has a particular format so we will probably never get there? Have you tried reading to EOF verifying it is all goodness you are getting back (i'm guessing you have)? We will get one of these for every file under a RS? 79 LOG.info(Allocating + StringUtils.byteDesc(fileSize) 80 + , on the path: + filePath); Could be a bunch. Maybe DEBUG? Though it would be good to have a message that verifies the filechannel mmap is working... so just leave as is... if it is annoying, can fix in a new JIRA Patch looks great. +1 pending answer to above question. Needs nice fat release note. I can add to the refguide too on commit. mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Attachments: ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13229) Specify bash for local-regionservers.sh and local-master-backup.sh
[
https://issues.apache.org/jira/browse/HBASE-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365850#comment-14365850
]
Hudson commented on HBASE-13229:
FAILURE: Integrated in HBase-0.94-JDK7 #231 (See
[https://builds.apache.org/job/HBase-0.94-JDK7/231/])
HBASE-13229 Specify bash for local-regionservers.sh and local-master-backup.sh
(busbey: rev 36c8f075f0b4c9e49f2fa3e2c7a80e0a921ac5a7)
* bin/local-regionservers.sh
* bin/local-master-backup.sh
Specify bash for local-regionservers.sh and local-master-backup.sh
--
Key: HBASE-13229
URL: https://issues.apache.org/jira/browse/HBASE-13229
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.94.0, 0.98.0, 1.0.0
Reporter: Gustavo Anatoly
Assignee: Gustavo Anatoly
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.94.27, 0.98.12
Attachments: HBASE-12339-v1.patch, HBASE-13229-v2.patch,
HBASE-13229.patch
Running the following line, using /bin/sh:
$ bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/ start 1 2
3 4 5
Produces the output below:
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
Considering:
{code}
if [[ $i =~ ^[0-9]+$ ]]; then
run_master $cmd $i
else
echo Invalid argument
fi
{code}
The reasons is that the regex operator =~ doesn't have compatibility with
/bin/sh but works running /bin/bash
$ bash -x bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/
start 1 2 3 4 5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zee Chen updated HBASE-13259: - Attachment: HBASE-13259.patch mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Attachments: HBASE-13259.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365888#comment-14365888
]
Josh Elser commented on HBASE-13262:
Appears to fix the trivial example I was trying to write.
So, it sounds like you're saying the fix/affects should also include 1.0.0, but
the true defect is still eluding us for the moment (I use us very loosely to
include my newbie-self)?
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365911#comment-14365911
]
Jonathan Lawlor commented on HBASE-13262:
-
I believe that 1.0.0 is also affected by the underlying issue (still not sure
what that actual cause is). When I ran your test on branch-1.0 with the Scan
configuration from branch-1+ (i.e. caching=Integer.Max_Value and
maxResultSize=2MB) I saw this at the output:
{quote}
2015-03-17 12:51:59,123 INFO [main] hbase.HBaseTest(167): Wrote 100 entries
2015-03-17 12:51:59,123 INFO [main] hbase.HBaseTest(182): Wrote 100
entries in total
2015-03-17 12:51:59,123 INFO [main] hbase.HBaseTest(185): Closing table used
for writes
2015-03-17 12:51:59,127 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '1\x00'
2015-03-17 12:51:59,327 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '2\x00'
2015-03-17 12:51:59,412 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '3\x00'
2015-03-17 12:51:59,477 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '4\x00'
2015-03-17 12:51:59,528 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '5\x00'
2015-03-17 12:51:59,567 INFO [main] hbase.HBaseTest(204): Saw row 500253
2015-03-17 12:51:59,597 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '6\x00'
2015-03-17 12:51:59,641 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '7\x00'
2015-03-17 12:51:59,683 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '8\x00'
2015-03-17 12:51:59,713 DEBUG [main] client.ClientScanner(260): Advancing
internal scanner to startKey at '9\x00'
2015-03-17 12:51:59,729 INFO [main] hbase.HBaseTest(204): Saw row 801761
2015-03-17 12:51:59,742 INFO [main] hbase.HBaseTest(211): Last row in Result
902495
2015-03-17 12:51:59,742 INFO [main] hbase.HBaseTest(214): Saw 23590 rows
2015-03-17 12:51:59,743 INFO [main] hbase.HBaseTest(215): Saw 235900 cells
2015-03-17 12:51:59,881 INFO [main] hbase.HBaseTest(220): Missing 976410
rows:...
{quote}
As [~elserj] called out above, it seems that we are jumping between regions too
early. This hypothesis is also supported by the fact that if you remove the
splits from table creation then the issue does not occur. Sorry, my intention
is not to hijack this issue, just trying to provide some information that may
lead us to the cause.
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13267) Deprecate or remove isFileDeletable from SnapshotHFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-13267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365970#comment-14365970 ] Mikhail Antonov commented on HBASE-13267: - Yeah. The only little concern here I could see is that having default implementation returning false makes it easier for implementers to forget to override the method (no compile-time check), hence preventing any files from being deleted. Maybe it isn't really important though. Deprecate or remove isFileDeletable from SnapshotHFileCleaner - Key: HBASE-13267 URL: https://issues.apache.org/jira/browse/HBASE-13267 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 The isFileDeletable method in SnapshotHFileCleaner became vestigial after HBASE-12627, lets remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13269) Limit result array preallocation to avoid OOME with large scan caching values
Andrew Purtell created HBASE-13269: -- Summary: Limit result array preallocation to avoid OOME with large scan caching values Key: HBASE-13269 URL: https://issues.apache.org/jira/browse/HBASE-13269 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.0.1, 0.98.12 Scan#setCaching(Integer.MAX_VALUE) will likely terminate the regionserver with an OOME due to preallocation of the result array according to this parameter. We should limit the preallocation to some sane value. Definitely affects 0.98 (fix needed to HRegionServer) and 1.0.x (fix needed to RsRPCServices), not sure about later versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13269) Limit result array preallocation to avoid OOME with large scan caching values
[ https://issues.apache.org/jira/browse/HBASE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366180#comment-14366180 ] Andrew Purtell commented on HBASE-13269: bq. otherwise it may just be best to remove the preallocation all together. This is exactly what I was thinking too and I'm testing out a patch now (although almost certainly every test will pass) Limit result array preallocation to avoid OOME with large scan caching values - Key: HBASE-13269 URL: https://issues.apache.org/jira/browse/HBASE-13269 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.0.1, 0.98.12 Scan#setCaching(Integer.MAX_VALUE) will likely terminate the regionserver with an OOME due to preallocation of the result array according to this parameter. We should limit the preallocation to some sane value. Definitely affects 0.98 (fix needed to HRegionServer) and 1.0.x (fix needed to RsRPCServices), not sure about later versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366108#comment-14366108 ] Zee Chen commented on HBASE-13259: -- btw the test result above is from hbase-0.98.10 patched running on a slightly modified version of JDK8. mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Fix For: 2.2.0 Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366128#comment-14366128
]
Sean Busbey commented on HBASE-13262:
-
sounds good to me. [~elserj] and [~jonathan.lawlor]? (y'all have better depth
on evaluating the issue)
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13269) Remove result array preallocation to avoid OOME with large scan caching values
[ https://issues.apache.org/jira/browse/HBASE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13269: --- Summary: Remove result array preallocation to avoid OOME with large scan caching values (was: Limit result array preallocation to avoid OOME with large scan caching values) Remove result array preallocation to avoid OOME with large scan caching values -- Key: HBASE-13269 URL: https://issues.apache.org/jira/browse/HBASE-13269 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.0.1, 0.98.12 Scan#setCaching(Integer.MAX_VALUE) will likely terminate the regionserver with an OOME due to preallocation of the result array according to this parameter. We should limit the preallocation to some sane value. Definitely affects 0.98 (fix needed to HRegionServer) and 1.0.x (fix needed to RsRPCServices), not sure about later versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13265) Make thrift2 usable from c++
[ https://issues.apache.org/jira/browse/HBASE-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365646#comment-14365646 ] Elliott Clark commented on HBASE-13265: --- bq.Is optional the default specifier? For unions one is required and the rest can't be there. So optional is kind of implied and kind wrong at the same time. Most versions of thrift this works just fine. On an internal build with a very recent version of thrift this breaks. bq.All these field name changes are to line up the IDL with the code, of will this require structural changes in the code as well? Including example build errors would give the rest of us context. There should be no code changes required since the parameters stayed the same. This should only change the generated code. Let my try and get the error from the internal customer who initially reported the issue. Make thrift2 usable from c++ Key: HBASE-13265 URL: https://issues.apache.org/jira/browse/HBASE-13265 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-13265.patch Currently the c++ code generated from our thrift2 idl doesn't compile. Mostly this is a naming issue for parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13147) Load actual META table descriptor, don't use statically defined one.
[ https://issues.apache.org/jira/browse/HBASE-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365813#comment-14365813 ] stack commented on HBASE-13147: --- +1 on master (again). We need extra checks added before we can add to branch-1. Load actual META table descriptor, don't use statically defined one. Key: HBASE-13147 URL: https://issues.apache.org/jira/browse/HBASE-13147 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Attachments: HBASE-13147-branch-1.patch, HBASE-13147-branch-1.v2.patch, HBASE-13147.patch, HBASE-13147.v2.patch, HBASE-13147.v3.patch, HBASE-13147.v4.patch, HBASE-13147.v4.patch In HBASE-13087 stumbled on the fact, that region servers don't see actual meta descriptor, they use their own, statically compiled. Need to fix that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13267) Deprecate or remove isFileDeletable from SnapshotHFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-13267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365812#comment-14365812 ] Mikhail Antonov commented on HBASE-13267: - Unless I'm missing something, this method is in BaseFileCleanerDelegate, so is a part of an interface? Deprecate or remove isFileDeletable from SnapshotHFileCleaner - Key: HBASE-13267 URL: https://issues.apache.org/jira/browse/HBASE-13267 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 The isFileDeletable method in SnapshotHFileCleaner became vestigial after HBASE-12627, lets remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zee Chen updated HBASE-13259: - Status: Patch Available (was: Open) mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Fix For: 2.2.0 Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zee Chen updated HBASE-13259: - Status: Open (was: Patch Available) mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Fix For: 2.2.0 Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366118#comment-14366118
]
Andrew Purtell commented on HBASE-13262:
Same result after https://github.com/apurtell/hbase-hwhat/commit/13385ef and a
similar hack to HRegionServer that Jonathan made to RSRpcServices on 1.0 to
remove result size preallocation.
2015-03-17 14:23:36,333 INFO [main] hbase.HBaseTest(167): Saw 100 rows
2015-03-17 14:23:36,333 INFO [main] hbase.HBaseTest(168): Saw 1000 cells
2015-03-17 14:23:36,556 INFO [main] hbase.HBaseTest(173): Missing 0 rows: []
Anything else I should try?
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366147#comment-14366147
]
Jonathan Lawlor commented on HBASE-13262:
-
May be worthwhile to do a sanity check to make sure that it breaks on
branch-1.0 for you as well. Otherwise I believe we need to dig deeper into the
internals. Currently investigating
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366083#comment-14366083
]
Andrew Purtell commented on HBASE-13262:
It's not a problem in 0.98 it seems.
I tested with this: https://github.com/apurtell/hbase-hwhat/tree/0.98
{noformat}
2015-03-17 14:00:52,911 WARN [main] util.NativeCodeLoader(62): Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2015-03-17 14:00:53,103 INFO [main] zookeeper.RecoverableZooKeeper(121):
Process identifier=hconnection-0x7afc920e connecting to ZooKeeper
ensemble=localhost:2181
2015-03-17 14:00:53,167 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(320): hconnection-0x7afc920e0x0,
quorum=localhost:2181, baseZNode=/hbase Received ZooKeeper Event, type=None,
state=SyncConnected, path=null
2015-03-17 14:00:53,171 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(397): hconnection-0x7afc920e-0x14c2985e89a0008
connected
2015-03-17 14:00:53,566 INFO [main] zookeeper.RecoverableZooKeeper(121):
Process identifier=hconnection-0x64198527 connecting to ZooKeeper
ensemble=localhost:2181
2015-03-17 14:00:53,574 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(320): hconnection-0x641985270x0,
quorum=localhost:2181, baseZNode=/hbase Received ZooKeeper Event, type=None,
state=SyncConnected, path=null
2015-03-17 14:00:53,578 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(397): hconnection-0x64198527-0x14c2985e89a0009
connected
2015-03-17 14:00:54,789 INFO [main] zookeeper.RecoverableZooKeeper(121):
Process identifier=catalogtracker-on-hconnection-0x64198527 connecting to
ZooKeeper ensemble=localhost:2181
2015-03-17 14:00:54,790 DEBUG [main] catalog.CatalogTracker(198): Starting
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3559b630
2015-03-17 14:00:54,797 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(320): catalogtracker-on-hconnection-0x641985270x0,
quorum=localhost:2181, baseZNode=/hbase Received ZooKeeper Event, type=None,
state=SyncConnected, path=null
2015-03-17 14:00:54,798 DEBUG [main-EventThread]
zookeeper.ZooKeeperWatcher(397):
catalogtracker-on-hconnection-0x64198527-0x14c2985e89a000a connected
2015-03-17 14:00:54,799 DEBUG [main] zookeeper.ZKUtil(430):
catalogtracker-on-hconnection-0x64198527-0x14c2985e89a000a,
quorum=localhost:2181, baseZNode=/hbase Set watcher on existing
znode=/hbase/meta-region-server
2015-03-17 14:00:54,821 DEBUG [main] catalog.CatalogTracker(222): Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3559b630
2015-03-17 14:00:54,837 INFO [main]
client.HConnectionManager$HConnectionImplementation(2246): Closing master
protocol: MasterService
2015-03-17 14:00:54,837 INFO [main]
client.HConnectionManager$HConnectionImplementation(1903): Closing zookeeper
sessionid=0x14c2985e89a0009
Write buffer size: 52428800
2015-03-17 14:01:05,010 INFO [main] hbase.HBaseTest(118): Wrote 5 entries
2015-03-17 14:01:12,491 INFO [main] hbase.HBaseTest(118): Wrote 10 entries
2015-03-17 14:01:19,860 INFO [main] hbase.HBaseTest(118): Wrote 15 entries
2015-03-17 14:01:27,223 INFO [main] hbase.HBaseTest(118): Wrote 20 entries
2015-03-17 14:01:34,772 INFO [main] hbase.HBaseTest(118): Wrote 25 entries
2015-03-17 14:01:41,784 INFO [main] hbase.HBaseTest(118): Wrote 30 entries
2015-03-17 14:01:49,003 INFO [main] hbase.HBaseTest(118): Wrote 35 entries
2015-03-17 14:01:56,044 INFO [main] hbase.HBaseTest(118): Wrote 40 entries
2015-03-17 14:02:03,331 INFO [main] hbase.HBaseTest(118): Wrote 45 entries
2015-03-17 14:02:10,360 INFO [main] hbase.HBaseTest(118): Wrote 50 entries
2015-03-17 14:02:18,349 INFO [main] hbase.HBaseTest(118): Wrote 55 entries
2015-03-17 14:02:26,238 INFO [main] hbase.HBaseTest(118): Wrote 60 entries
2015-03-17 14:02:33,430 INFO [main] hbase.HBaseTest(118): Wrote 65 entries
2015-03-17 14:02:40,362 INFO [main] hbase.HBaseTest(118): Wrote 70 entries
2015-03-17 14:02:47,358 INFO [main] hbase.HBaseTest(118): Wrote 75 entries
2015-03-17 14:02:54,396 INFO [main] hbase.HBaseTest(118): Wrote 80 entries
2015-03-17 14:03:01,735 INFO [main] hbase.HBaseTest(118): Wrote 85 entries
2015-03-17 14:03:08,691 INFO [main] hbase.HBaseTest(118): Wrote 90 entries
2015-03-17 14:03:16,234 INFO [main] hbase.HBaseTest(118): Wrote 95 entries
2015-03-17 14:03:22,987 INFO [main] hbase.HBaseTest(118): Wrote 100 entries
2015-03-17 14:03:22,987 INFO [main] hbase.HBaseTest(133): Wrote 100
entries in total
2015-03-17 14:03:22,987 INFO [main] hbase.HBaseTest(136): Closing table used
for writes
2015-03-17 14:03:23,000 DEBUG [main] client.ClientScanner(277): Advancing
internal scanner to startKey at '1\x00'
2015-03-17 14:03:23,667 INFO [main] hbase.HBaseTest(156): Saw row 108997
2015-03-17 14:03:24,013 INFO [main]
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366095#comment-14366095
]
Sean Busbey commented on HBASE-13262:
-
I don't see where the reconfiguration is done like Jonathan did on branch-1.0.
Is the cache behavior change not needed on 0.98?
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366101#comment-14366101
]
Andrew Purtell commented on HBASE-13262:
Ah, missed the comment on this issue where that was mentioned. Let me try it.
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366100#comment-14366100
]
Zee Chen commented on HBASE-13259:
--
Test results under following conditions:
- 22 byte key to 32 byte val map stored in a table, 16k hfile blocksize
- uniform key distribution, tested with gets from large number of client threads
- hbase.regionserver.handler.count=100
- hbase.bucketcache.size=7 (70GB)
- hbase.bucketcache.combinedcache.enabled=true
- hbase.bucketcache.ioengine=mmap:/dev/shm/bucketcache.0
-
hbase.bucketcache.bucket.sizes=5120,7168,9216,11264,13312,17408,33792,41984,50176,58368,66560,99328,132096,197632,263168,394240,525312
- CMS GC
At 85k gets per second, the system looks like:
{code}
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
58 11 26 0 0 5| 016k| 17M 13M| 0 0 | 316k 255k
59 11 25 0 0 5|2048k 12k| 18M 13M| 0 0 | 319k 254k
58 11 25 0 0 5| 028k| 18M 13M| 0 0 | 318k 253k
59 11 25 0 0 5|2048k0 | 18M 13M| 0 0 | 318k 252k
{code}
with wire latency profile (unit is microsecond):
{code}
Quantile: 0.50, Value: 361
Quantile: 0.75, Value: 555
Quantile: 0.90, Value: 830
Quantile: 0.95, Value: 1077
Quantile: 0.98, Value: 1604
Quantile: 0.99, Value: 4212
Quantile: 0.999000, Value: 7221
Quantile: 1.00, Value: 14406
{code}
FileIOEngine's latency profile is identical. It had higher sys CPU and lower
user CPU, higher context switches, and about 40% lower max throughput in gets
per second.
The patch was tested to 140k gets per second for 2 weeks nonstop.
mmap() based BucketCache IOEngine
-
Key: HBASE-13259
URL: https://issues.apache.org/jira/browse/HBASE-13259
Project: HBase
Issue Type: New Feature
Components: BlockCache
Affects Versions: 0.98.10
Reporter: Zee Chen
Fix For: 2.2.0
Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg,
mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data
from kernel space to user space. This is a good choice when the total working
set size is much bigger than the available RAM and the latency is dominated
by IO access. However, when the entire working set is small enough to fit in
the RAM, using mmap() (and subsequent memcpy()) to move data from kernel
space to user space is faster. I have run some short keyval gets tests and
the results indicate a reduction of 2%-7% of kernel CPU on my system,
depending on the load. On the gets, the latency histograms from mmap() are
identical to those from pread(), but peak throughput is close to 40% higher.
This patch modifies ByteByfferArray to allow it to specify a backing file.
Example for using this feature: set hbase.bucketcache.ioengine to
mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
Attached perf measured CPU usage breakdown in flames graph.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366096#comment-14366096
]
Jonathan Lawlor commented on HBASE-13262:
-
I was also unable to reproduce the issue in 0.98; saw the same output as
[~apurtell]
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366112#comment-14366112
]
Mikhail Antonov commented on HBASE-13103:
-
Yep, that's how I see it too.
Practically, I think, splitting of excessively large regions (in environments
with very high max region size or pre-split policy in place) would be more
frequent usecase, than merging, since tables tend to grow up over time, not
shrink down.
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Brainstorming
Components: Usability
Reporter: Nick Dimiduk
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366126#comment-14366126
]
Andrew Purtell commented on HBASE-13262:
bq. Note that to run the scan with this configuraiton you must remove the
ArrayList presizing that is performed in RSRpcServices on line 2055. The
caching value is used to presize the array and thus you will likely run into
OOME if you do not remove the presizing when running this configuration
That's an issue in 0.98 too, see HBASE-13269 to follow up
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13269) Limit result array preallocation to avoid OOME with large scan caching values
[ https://issues.apache.org/jira/browse/HBASE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366142#comment-14366142 ] Jonathan Lawlor commented on HBASE-13269: - After a quick discussion in HBASE-11544 the preallocation was removed because of this issue (there didn't seem to be any way to accurately estimate the number of Results that will be returned since the caching and result size limits work together). This means that in branch-1+ there is no preallocation (this change should have been backported further to prevent the OOM in earlier versions) If a more sophisticated solution is found here, then that change could be added to branch-1+ (HBASE-11544 is only in branch-1+) otherwise it may just be best to remove the preallocation all together. Limit result array preallocation to avoid OOME with large scan caching values - Key: HBASE-13269 URL: https://issues.apache.org/jira/browse/HBASE-13269 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 1.0.1, 0.98.12 Scan#setCaching(Integer.MAX_VALUE) will likely terminate the regionserver with an OOME due to preallocation of the result array according to this parameter. We should limit the preallocation to some sane value. Definitely affects 0.98 (fix needed to HRegionServer) and 1.0.x (fix needed to RsRPCServices), not sure about later versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366193#comment-14366193
]
Nick Dimiduk commented on HBASE-13103:
--
I've been thinking a lot about this feature in the context of a user who's
upgrading from 0.92 or 0.94 with little 1-2g regions. Suddenly they have way
more regions than necessary, so the region merge becomes very useful.
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Brainstorming
Components: Usability
Reporter: Nick Dimiduk
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12908) Typos in MemStoreFlusher javadocs
[ https://issues.apache.org/jira/browse/HBASE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364854#comment-14364854 ] Hudson commented on HBASE-12908: SUCCESS: Integrated in HBase-1.1 #296 (See [https://builds.apache.org/job/HBase-1.1/296/]) HBASE-12908 Typos in MemstoreFlusher javadocs (Edvin Malinovskis) (jmhsieh: rev 489698d6c02ae65270ad38aacc69fbe0030336b1) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java Typos in MemStoreFlusher javadocs - Key: HBASE-12908 URL: https://issues.apache.org/jira/browse/HBASE-12908 Project: HBase Issue Type: Bug Components: documentation Reporter: Edvin Malinovskis Priority: Trivial Labels: javadoc, typo Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-12908.patch The javadocs for the `flushRegion` methods say that _there will be accompanying log messages explaining why the *log* was not flushed._; when it should say _... why the *region* was not flushed._ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12908) Typos in MemStoreFlusher javadocs
[ https://issues.apache.org/jira/browse/HBASE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364853#comment-14364853 ] Hudson commented on HBASE-12908: SUCCESS: Integrated in HBase-1.0 #808 (See [https://builds.apache.org/job/HBase-1.0/808/]) HBASE-12908 Typos in MemstoreFlusher javadocs (Edvin Malinovskis) (jmhsieh: rev 212cba16fb427c0e2a77ea2ecaa08e5f125a7d7f) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java Typos in MemStoreFlusher javadocs - Key: HBASE-12908 URL: https://issues.apache.org/jira/browse/HBASE-12908 Project: HBase Issue Type: Bug Components: documentation Reporter: Edvin Malinovskis Priority: Trivial Labels: javadoc, typo Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-12908.patch The javadocs for the `flushRegion` methods say that _there will be accompanying log messages explaining why the *log* was not flushed._; when it should say _... why the *region* was not flushed._ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13265) Make thrift2 usable from c++
[ https://issues.apache.org/jira/browse/HBASE-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365640#comment-14365640 ] Andrey Stepachev commented on HBASE-13265: -- [~ndimiduk] looks good to me. union assumes optional, so it is ok for this change. Make thrift2 usable from c++ Key: HBASE-13265 URL: https://issues.apache.org/jira/browse/HBASE-13265 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.1.0 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-13265.patch Currently the c++ code generated from our thrift2 idl doesn't compile. Mostly this is a naming issue for parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365687#comment-14365687 ] Andrew Purtell commented on HBASE-13259: [~ndimiduk], you expressed interest in this on dev@ mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Attachments: ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13267) Deprecate or remove isFileDeletable from SnapshotHFileCleaner
Andrew Purtell created HBASE-13267: -- Summary: Deprecate or remove isFileDeletable from SnapshotHFileCleaner Key: HBASE-13267 URL: https://issues.apache.org/jira/browse/HBASE-13267 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 The isFileDeletable method in SnapshotHFileCleaner became vestigial after HBASE-12627, lets remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13267) Deprecate or remove isFileDeletable from SnapshotHFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-13267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365716#comment-14365716 ] Andrew Purtell commented on HBASE-13267: Any concerns? Deprecate or remove isFileDeletable from SnapshotHFileCleaner - Key: HBASE-13267 URL: https://issues.apache.org/jira/browse/HBASE-13267 Project: HBase Issue Type: Task Reporter: Andrew Purtell Priority: Minor Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 The isFileDeletable method in SnapshotHFileCleaner became vestigial after HBASE-12627, lets remove it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13262:
-
Affects Version/s: 1.1.0
2.0.0
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365826#comment-14365826 ] Zee Chen commented on HBASE-13259: -- ByteBufferArray has DEFAULT_BUFFER_SIZE set to 4MB right now. For realistic deployment scenarios, where regionservers can take 100+GB, we should allow this to be set to 1GB (see https://wiki.debian.org/Hugepages#x86_64 ) if the underlying system is configured for it. mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Attachments: ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365666#comment-14365666
]
Josh Elser commented on HBASE-13262:
Bisect'ed back to HBASE-11544 as the cause.
{noformat}
0c64a57e529d591a96d56721a1ae538167a03a11 is the first bad commit
commit 0c64a57e529d591a96d56721a1ae538167a03a11
Author: Jonathan Lawlor jonathan.law...@cloudera.com
Date: Wed Feb 4 14:07:35 2015 -0800
HBASE-11544: [Ergonomics] hbase.client.scanner.caching is dogged and will
try to return batch even if it means OOME
Signed-off-by: stack st...@apache.org
{noformat}
Going to try to investigate given the changeset.
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13126) Clean up API for unintended methods within non-private classes.
[ https://issues.apache.org/jira/browse/HBASE-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365718#comment-14365718 ] Sean Busbey commented on HBASE-13126: - This would be changing so that a not-yet-extant hbase-testing-utility jar is needed, since right now that module is just an aggregator. I haven't looked at implementing this yet, but I'd guess that hbase-server-test would go away. Clean up API for unintended methods within non-private classes. --- Key: HBASE-13126 URL: https://issues.apache.org/jira/browse/HBASE-13126 Project: HBase Issue Type: Task Components: API Affects Versions: 2.0.0 Reporter: Sean Busbey Priority: Blocker Fix For: 2.0.0 Over in the review for HBASE-12972, [~enis] mentioned that one of the HBTU methods wasn't intended for public consumption. Can we build a list of such methods across the API, appropriately annotate them for 2.0, and deprecate them in earlier versions with a warning that they're going to be restricted? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13266) test-patch.sh can return false positives for zombie tests from tests running on the same host
Esteban Gutierrez created HBASE-13266:
-
Summary: test-patch.sh can return false positives for zombie tests
from tests running on the same host
Key: HBASE-13266
URL: https://issues.apache.org/jira/browse/HBASE-13266
Project: HBase
Issue Type: Bug
Reporter: Esteban Gutierrez
Just saw this here
https://builds.apache.org/job/PreCommit-HBASE-Build/13271//consoleFull
{code}
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:27 h
[INFO] Finished at: 2015-03-16T23:58:30+00:00
[INFO] Final Memory: 93M/844M
[INFO]
Suspicious java process found - waiting 30s to see if there are just slow to
stop
There are 1 zombie tests, they should have been killed by surefire but survived
BEGIN zombies jstack extract
2015-03-16 23:59:03
Full thread dump Java HotSpot(TM) Server VM (23.25-b01 mixed mode):
Attach Listener daemon prio=10 tid=0xaa400800 nid=0x17cc waiting on condition
[0x]
java.lang.Thread.State: RUNNABLE
IPC Client (47) connection to 0.0.0.0/0.0.0.0:4324 from jenkins daemon
prio=10 tid=0xa8d03400 nid=0x1759 in Object.wait() [0xa9c7d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xde1987c8 (a org.apache.hama.ipc.Client$Connection)
at org.apache.hama.ipc.Client$Connection.waitForWork(Client.java:533)
- locked 0xde1987c8 (a org.apache.hama.ipc.Client$Connection)
at org.apache.hama.ipc.Client$Connection.run(Client.java:577)
...
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hama.bsp.TestBSPTaskFaults.tearDown(TestBSPTaskFaults.java:618)
at junit.framework.TestCase.runBare(TestCase.java:140)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
{code}
Which is getting a jstack from a test from Hama:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-13229) Specify bash for local-regionservers.sh and local-master-backup.sh
[
https://issues.apache.org/jira/browse/HBASE-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-13229.
-
Resolution: Fixed
re-pushed to all branches. Thanks for the suggestion Nick.
Specify bash for local-regionservers.sh and local-master-backup.sh
--
Key: HBASE-13229
URL: https://issues.apache.org/jira/browse/HBASE-13229
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.94.0, 0.98.0, 1.0.0
Reporter: Gustavo Anatoly
Assignee: Gustavo Anatoly
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.94.27, 0.98.12
Attachments: HBASE-12339-v1.patch, HBASE-13229-v2.patch,
HBASE-13229.patch
Running the following line, using /bin/sh:
$ bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/ start 1 2
3 4 5
Produces the output below:
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
Considering:
{code}
if [[ $i =~ ^[0-9]+$ ]]; then
run_master $cmd $i
else
echo Invalid argument
fi
{code}
The reasons is that the regex operator =~ doesn't have compatibility with
/bin/sh but works running /bin/bash
$ bash -x bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/
start 1 2 3 4 5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365815#comment-14365815
]
Enis Soztutar commented on HBASE-13262:
---
bq. Specifically, the following configuration also produced issues for me in
branch-1.0
HBASE-11544 is not in 1.0 branch. Is this a different issue? Should we create a
subtask and fix it in 1.0.1?
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13147) Load actual META table descriptor, don't use statically defined one.
[
https://issues.apache.org/jira/browse/HBASE-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365579#comment-14365579
]
Hadoop QA commented on HBASE-13147:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705091/HBASE-13147.v4.patch
against master branch at commit ed026c2e6f518963ee7fb9c7b26266e888c71c2d.
ATTACHMENT ID: 12705091
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1919 checkstyle errors (more than the master's current 1917 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13285//console
This message is automatically generated.
Load actual META table descriptor, don't use statically defined one.
Key: HBASE-13147
URL: https://issues.apache.org/jira/browse/HBASE-13147
Project: HBase
Issue Type: Bug
Components: master, regionserver
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Attachments: HBASE-13147-branch-1.patch,
HBASE-13147-branch-1.v2.patch, HBASE-13147.patch, HBASE-13147.v2.patch,
HBASE-13147.v3.patch, HBASE-13147.v4.patch, HBASE-13147.v4.patch
In HBASE-13087 stumbled on the fact, that region servers don't see actual
meta descriptor, they use their own, statically compiled.
Need to fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12990) MetaScanner should be replaced by MetaTableAccessor
[
https://issues.apache.org/jira/browse/HBASE-12990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365773#comment-14365773
]
Hadoop QA commented on HBASE-12990:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705110/HBASE-12990-branch-1.v1.patch
against branch-1 branch at commit ca8876a9f2534f2ab0b416aecb4ac476da9747f8.
ATTACHMENT ID: 12705110
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 59 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3810 checkstyle errors (more than the master's current 3807 errors).
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint
org.apache.hadoop.hbase.replication.TestMasterReplication
org.apache.hadoop.hbase.filter.TestScanRowPrefix
org.apache.hadoop.hbase.mapred.TestTableInputFormat
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.coprocessor.TestBatchCoprocessorEndpoint
org.apache.hadoop.hbase.client.TestAdmin2
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.TestMultiVersions
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
org.apache.hadoop.hbase.client.TestHTableMultiplexer
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.regionserver.TestZKLessSplitOnCluster
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload
org.apache.hadoop.hbase.util.TestRegionSplitter
org.apache.hadoop.hbase.filter.TestFilterWithScanLimits
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable
org.apache.hadoop.hbase.client.TestHTablePool$TestHTableReusablePool
org.apache.hadoop.hbase.client.TestHTableMultiplexerFlushCache
org.apache.hadoop.hbase.TestMetaTableAccessorNoCluster
org.apache.hadoop.hbase.client.TestAdmin1
org.apache.hadoop.hbase.client.TestHTablePool$TestHTableThreadLocalPool
org.apache.hadoop.hbase.client.TestReplicaWithCluster
org.apache.hadoop.hbase.master.handler.TestEnableTableHandler
org.apache.hadoop.hbase.util.TestMergeTable
org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver
org.apache.hadoop.hbase.TestMetaTableAccessor
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
org.apache.hadoop.hbase.filter.TestFilterWrapper
org.apache.hadoop.hbase.mapreduce.TestCopyTable
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestCompactionState
[jira] [Commented] (HBASE-13093) Local mode HBase instance doesn't shut down.
[
https://issues.apache.org/jira/browse/HBASE-13093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365594#comment-14365594
]
Hudson commented on HBASE-13093:
FAILURE: Integrated in HBase-1.1 #298 (See
[https://builds.apache.org/job/HBase-1.1/298/])
HBASE-13093 Local mode HBase instance doesn't shut down. (octo47: rev
d0af30ea3cc33c970c323917d6d522b4536071da)
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcClient.java
Local mode HBase instance doesn't shut down.
Key: HBASE-13093
URL: https://issues.apache.org/jira/browse/HBASE-13093
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Elliott Clark
Assignee: Andrey Stepachev
Fix For: 2.0.0, 1.1.1
Attachments: HBASE-13093.patch, HBASE-13093.v2.patch,
HBASE-13093.v2.patch, HBASE-13093.v2.patch, HBASE-13093.v2.patch,
HBASE-13093.v2.patch
{code}bin/start-hbase.sh{code}
{code}bin/stop-hbase.sh{code}
That hangs forever. Here's the jstacks:
{code}2015-02-24 16:37:55
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode):
Attach Listener daemon prio=5 tid=0x7fb130813800 nid=0xfd07 waiting on
condition [0x]
java.lang.Thread.State: RUNNABLE
DestroyJavaVM prio=5 tid=0x7fb12ba7c800 nid=0x1303 waiting on condition
[0x]
java.lang.Thread.State: RUNNABLE
pool-5-thread-1 prio=5 tid=0x7fb12bb88800 nid=0x19903 waiting on
condition [0x000121a1b000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360)
at java.lang.Thread.run(Thread.java:745)
HBase-Metrics2-1 daemon prio=5 tid=0x7fb12c04 nid=0x19703 waiting
on condition [0x000121918000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000724cc9780 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
snapshot-hfile-cleaner-cache-refresher daemon prio=5 tid=0x7fb12bc91000
nid=0x18703 in Object.wait() [0x00012160f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0x000724caa588 (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked 0x000724caa588 (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
snapshot-log-cleaner-cache-refresher daemon prio=5 tid=0x7fb12bbc8000
nid=0x18503 in Object.wait() [0x00012150c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0x000724deb178 (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked 0x000724deb178 (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
localhost:57343.activeMasterManager-EventThread daemon prio=5
tid=0x7fb12c072000 nid=0x18303 waiting on condition [0x000121409000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x000724f10150 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
localhost:57343.activeMasterManager-SendThread(fe80:0:0:0:0:0:0:1%1:2181)
daemon prio=5 tid=0x7fb12c053000 nid=0x18103 waiting on condition
[jira] [Commented] (HBASE-13176) Flakey TestZooKeeper test.
[
https://issues.apache.org/jira/browse/HBASE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365593#comment-14365593
]
Hudson commented on HBASE-13176:
FAILURE: Integrated in HBase-1.1 #298 (See
[https://builds.apache.org/job/HBase-1.1/298/])
HBASE-13176 Flakey TestZooKeeper test. (octo47: rev
bee9fb8e7e3a0c3dcdc631908a09fbd4bc11efb0)
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
Flakey TestZooKeeper test.
--
Key: HBASE-13176
URL: https://issues.apache.org/jira/browse/HBASE-13176
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Fix For: 2.0.0, 1.0.1, 1.1.1
Attachments: HBASE-13176.patch
Test fails with wrong number of listeners:
{code}
Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 202.248 sec
FAILURE! - in org.apache.hadoop.hbase.TestZooKeeper
testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(org.apache.hadoop.hbase.TestZooKeeper)
Time elapsed: 48.687 sec FAILURE!
java.lang.AssertionError: expected:16 but was:15
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.TestZooKeeper.testRegionAssignmentAfterMasterRecoveryDueToZKExpiry(TestZooKeeper.java:518)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13257: Assignee: zhangduo Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13266) test-patch.sh can return false positives for zombie tests from tests running on the same host
[
https://issues.apache.org/jira/browse/HBASE-13266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365810#comment-14365810
]
stack commented on HBASE-13266:
---
Yeah. We've been seeing this. What would you recommend Mr. Regex?
Here is what we have up in jenkins to run post build:
{code}
ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
#It seems sometimes the tests are not dying immediately. Let's give them 10s
echo Suspicious java process found - waiting 10s to see if there are just
slow to stop
sleep 10
ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
echo There are $ZOMBIE_TESTS_COUNT zombie tests, they should have been
killed by surefire but survived
echo BEGIN zombies jstack extract
ZB_STACK=`jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack
| grep .test | grep \.java`
jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack
echo END zombies jstack extract
JIRA_COMMENT=$JIRA_COMMENT
{color:red}-1 core zombie tests{color}. There are ${ZOMBIE_TESTS_COUNT}
zombie test(s): ${ZB_STACK}
BAD=1
jps | grep surefirebooter | cut -d ' ' -f 1 | xargs kill -9
else
echo We're ok: there is no zombie test, but some tests took some time to
stop
fi
else
echo We're ok: there is no zombie test
fi
{code}
... add a '-ei' and search for hbase?
test-patch.sh can return false positives for zombie tests from tests running
on the same host
-
Key: HBASE-13266
URL: https://issues.apache.org/jira/browse/HBASE-13266
Project: HBase
Issue Type: Bug
Reporter: Esteban Gutierrez
Just saw this here
https://builds.apache.org/job/PreCommit-HBASE-Build/13271//consoleFull
{code}
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 01:27 h
[INFO] Finished at: 2015-03-16T23:58:30+00:00
[INFO] Final Memory: 93M/844M
[INFO]
Suspicious java process found - waiting 30s to see if there are just slow to
stop
There are 1 zombie tests, they should have been killed by surefire but
survived
BEGIN zombies jstack extract
2015-03-16 23:59:03
Full thread dump Java HotSpot(TM) Server VM (23.25-b01 mixed mode):
Attach Listener daemon prio=10 tid=0xaa400800 nid=0x17cc waiting on
condition [0x]
java.lang.Thread.State: RUNNABLE
IPC Client (47) connection to 0.0.0.0/0.0.0.0:4324 from jenkins daemon
prio=10 tid=0xa8d03400 nid=0x1759 in Object.wait() [0xa9c7d000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0xde1987c8 (a org.apache.hama.ipc.Client$Connection)
at org.apache.hama.ipc.Client$Connection.waitForWork(Client.java:533)
- locked 0xde1987c8 (a org.apache.hama.ipc.Client$Connection)
at org.apache.hama.ipc.Client$Connection.run(Client.java:577)
...
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.hama.bsp.TestBSPTaskFaults.tearDown(TestBSPTaskFaults.java:618)
at junit.framework.TestCase.runBare(TestCase.java:140)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
{code}
Which is getting a jstack from a test from Hama.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13265) Make thrift2 usable from c++
[
https://issues.apache.org/jira/browse/HBASE-13265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365617#comment-14365617
]
Nick Dimiduk commented on HBASE-13265:
--
{noformat}
union TMutation {
- 1: optional TPut put,
- 2: optional TDelete deleteSingle,
+ 1: TPut put,
+ 2: TDelete deleteSingle,
}
{noformat}
Is {{optional}} the default specifier?
All these field name changes are to line up the IDL with the code, of will this
require structural changes in the code as well? Including example build errors
would give the rest of us context.
+1 if this works for you Elliott.
[~larsgeorge], [~octo47] want to have a look?
Make thrift2 usable from c++
Key: HBASE-13265
URL: https://issues.apache.org/jira/browse/HBASE-13265
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 1.1.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-13265.patch
Currently the c++ code generated from our thrift2 idl doesn't compile. Mostly
this is a naming issue for parameters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13262:
-
Fix Version/s: 2.0.0
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13229) Specify bash for local-regionservers.sh and local-master-backup.sh
[
https://issues.apache.org/jira/browse/HBASE-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365831#comment-14365831
]
Hudson commented on HBASE-13229:
SUCCESS: Integrated in HBase-0.94-security #576 (See
[https://builds.apache.org/job/HBase-0.94-security/576/])
HBASE-13229 Specify bash for local-regionservers.sh and local-master-backup.sh
(busbey: rev 36c8f075f0b4c9e49f2fa3e2c7a80e0a921ac5a7)
* bin/local-master-backup.sh
* bin/local-regionservers.sh
Specify bash for local-regionservers.sh and local-master-backup.sh
--
Key: HBASE-13229
URL: https://issues.apache.org/jira/browse/HBASE-13229
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.94.0, 0.98.0, 1.0.0
Reporter: Gustavo Anatoly
Assignee: Gustavo Anatoly
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.94.27, 0.98.12
Attachments: HBASE-12339-v1.patch, HBASE-13229-v2.patch,
HBASE-13229.patch
Running the following line, using /bin/sh:
$ bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/ start 1 2
3 4 5
Produces the output below:
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
Considering:
{code}
if [[ $i =~ ^[0-9]+$ ]]; then
run_master $cmd $i
else
echo Invalid argument
fi
{code}
The reasons is that the regex operator =~ doesn't have compatibility with
/bin/sh but works running /bin/bash
$ bash -x bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/
start 1 2 3 4 5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13269) Remove result array preallocation to avoid OOME with large scan caching values
[
https://issues.apache.org/jira/browse/HBASE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366486#comment-14366486
]
Hadoop QA commented on HBASE-13269:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12705206/HBASE-13269-0.98.patch
against 0.98 branch at commit f9a17edc252a88c5a1a2c7764e3f9f65623e0ced.
ATTACHMENT ID: 12705206
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
25 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to introduce 8 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/13291//console
This message is automatically generated.
Remove result array preallocation to avoid OOME with large scan caching values
--
Key: HBASE-13269
URL: https://issues.apache.org/jira/browse/HBASE-13269
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 1.0.1, 0.98.12
Attachments: HBASE-13269-0.98.patch, HBASE-13269-1.0.patch
Scan#setCaching(Integer.MAX_VALUE) will likely terminate the regionserver
with an OOME due to preallocation of the result array according to this
parameter. We should limit the preallocation to some sane value. Definitely
affects 0.98 (fix needed to HRegionServer) and 1.0.x (fix needed to
RsRPCServices), not sure about later versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13200) Improper configuration can leads to endless lease recovery during failover
[ https://issues.apache.org/jira/browse/HBASE-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366510#comment-14366510 ] Liu Shaohui commented on HBASE-13200: - +1 for me. Improper configuration can leads to endless lease recovery during failover -- Key: HBASE-13200 URL: https://issues.apache.org/jira/browse/HBASE-13200 Project: HBase Issue Type: Bug Components: MTTR Reporter: He Liangliang Assignee: He Liangliang Attachments: HBASE-13200.patch When a node (DN+RS) has machine/OS level failure, another RS will try to do lease recovery for the log file. It will retry for every hbase.lease.recovery.dfs.timeout (default to 61s) from the second time. When the hdfs configuration is not properly configured (e.g. socket connection timeout) and without patch HDFS-4721, the lease recovery time can exceeded the timeout specified by hbase.lease.recovery.dfs.timeout. This will lead to endless retries and preemptions until the final timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366521#comment-14366521 ] Sean Busbey commented on HBASE-13257: - I have gotten spoiled by the cobertura plugin's ability to show code coverage in a way that we can drill down to show coverage per package and per class. Can we configure the Jacoco plugin to do some of that reporting? Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366535#comment-14366535 ] Sean Busbey commented on HBASE-13257: - yes, perfect! Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13262) ResultScanner doesn't return all rows in Scan
[
https://issues.apache.org/jira/browse/HBASE-13262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366436#comment-14366436
]
Josh Elser commented on HBASE-13262:
bq. So within RSRpcServices#scan(...) we keep a running tally of the size of
the accumulated Result within the variable currentScanResultSize. We collect
the Result in a while loop that loops while the caching limit hasn't been
reached. At the beginning of each iteration of this loop, we check the running
Result size limit against the maxResultSize. If the size limit has been
reached, we break out of the loop and will end up returning whatever Results we
have accumulated thus far back to the client. The problem is that we then
expect the Client to realize that the Results they receive are larger than the
maxResultSize – if the client's size calculation is less than the server's then
it's possible the client will misinterpret the response as meaning the region
has been exhausted.
Thanks for the pointer. I will investigate.
ResultScanner doesn't return all rows in Scan
-
Key: HBASE-13262
URL: https://issues.apache.org/jira/browse/HBASE-13262
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 2.0.0, 1.1.0
Environment: Single node, pseduo-distributed 1.1.0-SNAPSHOT
Reporter: Josh Elser
Assignee: Josh Elser
Priority: Blocker
Fix For: 2.0.0, 1.1.0
Attachments: testrun_0.98.txt, testrun_branch1.0.txt
Tried to write a simple Java client again 1.1.0-SNAPSHOT.
* Write 1M rows, each row with 1 family, and 10 qualifiers (values [0-9]),
for a total of 10M cells written
* Read back the data from the table, ensure I saw 10M cells
Running it against {{04ac1891}} (and earlier) yesterday, I would get ~20% of
the actual rows. Running against 1.0.0, returns all 10M records as expected.
[Code I was
running|https://github.com/joshelser/hbase-hwhat/blob/master/src/main/java/hbase/HBaseTest.java]
for the curious.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13229) Specify bash for local-regionservers.sh and local-master-backup.sh
[
https://issues.apache.org/jira/browse/HBASE-13229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366509#comment-14366509
]
Hudson commented on HBASE-13229:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #861 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/861/])
HBASE-13229 Specify bash for local-regionservers.sh and local-master-backup.sh
(busbey: rev b428f10c392cadb011e19d0db495571d5d88f2b0)
* bin/local-regionservers.sh
* bin/local-master-backup.sh
Specify bash for local-regionservers.sh and local-master-backup.sh
--
Key: HBASE-13229
URL: https://issues.apache.org/jira/browse/HBASE-13229
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.94.0, 0.98.0, 1.0.0
Reporter: Gustavo Anatoly
Assignee: Gustavo Anatoly
Priority: Minor
Labels: beginner
Fix For: 2.0.0, 1.0.1, 1.1.0, 0.94.27, 0.98.12
Attachments: HBASE-12339-v1.patch, HBASE-13229-v2.patch,
HBASE-13229.patch
Running the following line, using /bin/sh:
$ bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/ start 1 2
3 4 5
Produces the output below:
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
bin/local-regionservers.sh: 55: bin/local-regionservers.sh: [[: not found
Invalid argument
Considering:
{code}
if [[ $i =~ ^[0-9]+$ ]]; then
run_master $cmd $i
else
echo Invalid argument
fi
{code}
The reasons is that the regex operator =~ doesn't have compatibility with
/bin/sh but works running /bin/bash
$ bash -x bin/local-regionservers.sh --config ~/hbase-dev/hbase-conf/conf/
start 1 2 3 4 5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13253) LoadIncrementalHFiles unify hfiles discovery
[ https://issues.apache.org/jira/browse/HBASE-13253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366508#comment-14366508 ] Hudson commented on HBASE-13253: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #861 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/861/]) HBASE-13253 LoadIncrementalHFiles unify hfiles discovery (matteo.bertozzi: rev a4d761a8a4d8a7577410eea514f52caf0f30) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java LoadIncrementalHFiles unify hfiles discovery Key: HBASE-13253 URL: https://issues.apache.org/jira/browse/HBASE-13253 Project: HBase Issue Type: Bug Components: Client, mapreduce Affects Versions: 1.0.0, 2.0.0, 1.1.0, 0.98.11 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0, 1.0.1, 1.1.0, 0.98.12 Attachments: HBASE-13253-v0.patch, HBASE-13253-v1.patch We have two copy-pasted code-path in createTable() and discoverLoadQueue(). They do does the same exact loop on the fs with the same validation logic. we should unify them, to avoid having them out of sync -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13257) Show coverage report on jenkins
[ https://issues.apache.org/jira/browse/HBASE-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366527#comment-14366527 ] zhangduo commented on HBASE-13257: -- You mean this? https://builds.apache.org/job/HBase-TRUNK-jacoco/4/jacoco/ Show coverage report on jenkins --- Key: HBASE-13257 URL: https://issues.apache.org/jira/browse/HBASE-13257 Project: HBase Issue Type: Task Reporter: zhangduo Assignee: zhangduo Priority: Minor Think of showing jacoco coverage report on https://builds.apache.org . And there is an advantage of showing it on jenkins that the jenkins jacoco plugin can handle cross module coverage. Can not do it locally since https://github.com/jacoco/jacoco/pull/97 is still pending. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13200) Improper configuration can leads to endless lease recovery during failover
[ https://issues.apache.org/jira/browse/HBASE-13200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366520#comment-14366520 ] Liu Shaohui commented on HBASE-13200: - Commit tomorrow if no objection. Improper configuration can leads to endless lease recovery during failover -- Key: HBASE-13200 URL: https://issues.apache.org/jira/browse/HBASE-13200 Project: HBase Issue Type: Bug Components: MTTR Reporter: He Liangliang Assignee: He Liangliang Attachments: HBASE-13200.patch When a node (DN+RS) has machine/OS level failure, another RS will try to do lease recovery for the log file. It will retry for every hbase.lease.recovery.dfs.timeout (default to 61s) from the second time. When the hdfs configuration is not properly configured (e.g. socket connection timeout) and without patch HDFS-4721, the lease recovery time can exceeded the timeout specified by hbase.lease.recovery.dfs.timeout. This will lead to endless retries and preemptions until the final timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13216) Add version info in RPC connection header
[ https://issues.apache.org/jira/browse/HBASE-13216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366519#comment-14366519 ] Liu Shaohui commented on HBASE-13216: - [~stack] Ok, i'll commit it tomorrow if no objection. Add version info in RPC connection header - Key: HBASE-13216 URL: https://issues.apache.org/jira/browse/HBASE-13216 Project: HBase Issue Type: Improvement Components: Client, rpc Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13216-v1.diff, HBASE-13216-v2.diff, HBASE-13216-v3.diff, HBASE-13216-v4.diff In the operation of a cluster, we usually want to know which clients are using the HBase client with critical bugs or too old version we will not support in future. By adding version info in RPC connection header, we can get these informations from audit log and promote them upgrade before a deadline. Discussions and suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
