[jira] [Commented] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions
[
https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618007#comment-13618007
]
Andrew Purtell commented on HBASE-8226:
---
I ran the trunk suite locally with this result:
{noformat}
Results :
Tests in error:
testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test
timed out after 60 milliseconds
Tests run: 2128, Failures: 0, Errors: 1, Skipped: 13
{noformat}
This looks like HBASE-8164.
Ran the 0.94 test suite locally and everything passed.
Going to commit this tomorrow.
> HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts
> "too many" regions
>
>
> Key: HBASE-8226
> URL: https://issues.apache.org/jira/browse/HBASE-8226
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.95.0, 0.96.0, 0.94.7
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Attachments: 8226-branch-0.94.patch, 8226.patch, 8226.patch
>
>
> HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region
> it is counting belongs to the table created by the test and will not return
> if it accidentally counts "too many" regions, for example the regions of the
> ACL table when security is enabled.
> Made an attempt at fixing this as part of HBASE-8209 but broke
> TestMasterTransitions. Try again here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8188) Avoid unnecessary row compare in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618006#comment-13618006 ] ramkrishna.s.vasudevan commented on HBASE-8188: --- Patch looks good. > Avoid unnecessary row compare in StoreScanner > - > > Key: HBASE-8188 > URL: https://issues.apache.org/jira/browse/HBASE-8188 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 8188-0.94.txt, 8188-0.96.txt, 8188-0.96.txt, > 8188-0.96.txt, 8188-0.96.txt > > > When scanner batching disable (which is the default) a row compare in > StoreScanner avoided, because only entire rows of data are requested. > This provides a slight performance gain, especially for tall tables (and when > all data is in the cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
[ https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618003#comment-13618003 ] Jieshan Bean commented on HBASE-8229: - I suggest to let ReplicationSource wait if one replicating table is not present, likes the scenario of peer cluster is unavailable. > Replication code logs like crazy if a target table cannot be found. > --- > > Key: HBASE-8229 > URL: https://issues.apache.org/jira/browse/HBASE-8229 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Fix For: 0.95.0, 0.98.0, 0.94.7 > > > One of our RS/DN machines ran out of diskspace on the partition to which we > write the log files. > It turns out we still had a table in our source cluster with > REPLICATION_SCOPE=>1 that did not have a matching table in the remote cluster. > In then logged a long stack trace every 50ms or so, over a few days that > filled up our log partition. > Since ReplicationSource cannot make any progress in this case anyway, it > should probably sleep a bit before retrying (or at least limit the rate at > which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618002#comment-13618002 ] ramkrishna.s.vasudevan commented on HBASE-5583: --- @Jimmy, When i was with my previous employer we actually wanted this feature. The reason was tables was dropped and created on a daily basis. So because of this problem, the developers had written a client code which creates the table by adding directly into META and then tries to enable those regions. I think that was in 0.90 code. The later version of hbase made the creation more reliable but often in our integration tests and Functional testing we end up with the tables getting half created and we end up in problems. That is why it prompted us to work on this. bq.s it simpler to disable the table and let the user create the table again in such a case Yes this is possible. But involves the following changes that i can think of immediately -> Even if the table znode is in ENABLING we should still disable the table. -> If we mark the table disabled, user should be very clear that the table he created is partial and that is what is DISABLED now. I think the better way is to use a new state- say UNUSABLE (or something like this) which will make the user to take a clear course of action. I still feel the above steps are simpler. But table creation is the basic step and handling the failure scenarios in this should be more of automated rather than user interference is what i felt. Hence decided to go on with this way. @Ted Thanks for going thro the patch. I will make the necessary changes and upload it. Just waiting for more ideas and views on this. > Master restart on create table with splitkeys does not recreate table with > all the splitkey regions > --- > > Key: HBASE-5583 > URL: https://issues.apache.org/jira/browse/HBASE-5583 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.95.0 > > Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_2.patch, > HBASE-5583_new_4_WIP.patch, HBASE-5583_new_5_WIP_using_tableznode.patch > > > -> Create table using splitkeys > -> MAster goes down before all regions are added to meta > -> On master restart the table is again enabled but with less number of > regions than specified in splitkeys > Anyway client will get an exception if i had called sync create table. But > table exists or not check will say table exists. > Is this scenario to be handled by client only or can we have some mechanism > on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8196) Port HBASE-7721 Atomic multi-row mutations in META to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8196: - Fix Version/s: (was: 0.94.7) > Port HBASE-7721 Atomic multi-row mutations in META to 0.94 > -- > > Key: HBASE-8196 > URL: https://issues.apache.org/jira/browse/HBASE-8196 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.7 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: hbase-7721_v4-0.94.patch > > > There has been some interest in porting region merge (HBASE-7403), and making > splits more stable in 0.94. For online merge, we depend on HBASE-7721, thus > we might need the backport. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
Lars Hofhansl created HBASE-8229: Summary: Replication code logs like crazy if a target table cannot be found. Key: HBASE-8229 URL: https://issues.apache.org/jira/browse/HBASE-8229 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 One of our RS/DN machines ran out of diskspace on the partition to which we write the log files. It turns out we still had a table in our source cluster with REPLICATION_SCOPE=>1 that did not have a matching table in the remote cluster. In then logged a long stack trace every 50ms or so, over a few days that filled up our log partition. Since ReplicationSource cannot make any progress in this case anyway, it should probably sleep a bit before retrying (or at least limit the rate at which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617992#comment-13617992 ] Hudson commented on HBASE-8164: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #469 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/469/]) HBASE-8164 TestTableLockManager fails intermittently in trunk builds, addendum (Revision 1462668) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestTableLockManager.java > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617993#comment-13617993 ] Hudson commented on HBASE-7904: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #469 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/469/]) HBASE-7904 Make mapreduce jobs pass based on 2.0.4-alpha (Revision 1462686) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * /hbase/trunk/pom.xml > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, > 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, > 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, > 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, > 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617994#comment-13617994 ] Hudson commented on HBASE-8222: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #469 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/469/]) HBASE-8222 User class should implement equals() and hashCode() (Revision 1462672) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUser.java > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8228) Investigate time taken to snapshot memstore
[ https://issues.apache.org/jira/browse/HBASE-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617991#comment-13617991 ] Himanshu Vashishtha commented on HBASE-8228: Interesting. How frequent? Any specific patterns... like while intensive write workload? When do you start its timer? Since taking the updateLock? > Investigate time taken to snapshot memstore > --- > > Key: HBASE-8228 > URL: https://issues.apache.org/jira/browse/HBASE-8228 > Project: HBase > Issue Type: Sub-task >Reporter: Amitanand Aiyer >Assignee: Amitanand Aiyer >Priority: Minor > Fix For: 0.89-fb > > > Snapshotting memstores is normally quick. But, sometimes it seems to take > long. This JIRA is to track the investigation and fix to improve the outliers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[ https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617989#comment-13617989 ] stack commented on HBASE-8224: -- [~enis] We have 11 modules of which at least 5 would need the shadow module you suggest. That is a bunch of modules. And then there is hadoop3 one day? I tried again to make the classifiers work (if they worked, it'd be 'elegant' -- caveat we are talking maven world here). I tried adding the dependencies to hbase-assembly so we could then do dependencySet in the assembly descriptor. I could get the hbase-*-hadoop?.jars to show up but they would come w/o their dependencies. I tried getting arty w/ maven coordinate specifications and dependencySet but it started to turn erratic on me. Things I've learned (not sure if they are hard and fast rules -- but I have not been able to get past them): + We cannot read the pom.version off the command-line. It has to be hard-coded in the top-level pom and in all submodules. I cannot interpolate a property into the pom.version either (to add hadoop1 or hadoop2 suffix); mvn just complains. + We could add hadoop? to the version but there is no connection between version and hadoop profile so we could have a hadoop1 added to jar names though we build the assembly with the hadoop2 profile (Maybe this is not too bad Serves you right if you end up w/ this combo). How we going to change the version though given the above? Making the assembly package, its not too bad. Could just name the assembly/package for the hadoop version and leave the jar alone unadorned with hadoop version. Publishing to maven we'd do something crazy like change the version number, commit, build and upload for hadoop1, then commit again, build, and upload for hadoop2? Any other ideas? > Add '-hadoop1' or '-hadoop2' to our version string > -- > > Key: HBASE-8224 > URL: https://issues.apache.org/jira/browse/HBASE-8224 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: 8224-adding.classifiers.txt > > > So we can publish both the hadoop1 and the hadoop2 jars to a maven > repository, and so we can publish two packages, one for hadoop1 and one for > hadoop2, given how maven works, our only alternative (to the best of my > knowledge and after consulting others) is by amending the version string to > include hadoop1 or hadoop2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8188) Avoid unnecessary row compare in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617985#comment-13617985
]
Hadoop QA commented on HBASE-8188:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576218/8188-0.96.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5063//console
This message is automatically generated.
> Avoid unnecessary row compare in StoreScanner
> -
>
> Key: HBASE-8188
> URL: https://issues.apache.org/jira/browse/HBASE-8188
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Attachments: 8188-0.94.txt, 8188-0.96.txt, 8188-0.96.txt,
> 8188-0.96.txt, 8188-0.96.txt
>
>
> When scanner batching disable (which is the default) a row compare in
> StoreScanner avoided, because only entire rows of data are requested.
> This provides a slight performance gain, especially for tall tables (and when
> all data is in the cache)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617984#comment-13617984 ] Hudson commented on HBASE-8222: --- Integrated in hbase-0.95-on-hadoop2 #48 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/48/]) HBASE-8222 User class should implement equals() and hashCode() (Revision 1462673) Result = FAILURE tedyu : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUser.java > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617983#comment-13617983 ] Hudson commented on HBASE-8164: --- Integrated in hbase-0.95-on-hadoop2 #48 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/48/]) HBASE-8164 TestTableLockManager fails intermittently in trunk builds, addendum (Revision 1462669) Result = FAILURE tedyu : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestTableLockManager.java > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7904: -- Attachment: 7904-addendum.txt > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, > 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, > 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, > 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, > 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7904: -- Status: Open (was: Patch Available) > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, > 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, > 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, > 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, > 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-7817:
---
Assignee: Vasu Mariyala
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.4
>Reporter: Ian Varley
>Assignee: Vasu Mariyala
>Priority: Minor
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: HBASE-7817.patch
>
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8174) Allow each table to customize its own flush blocking file count "hbase.hstore.blockingStoreFiles"
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617980#comment-13617980 ] Lars Hofhansl commented on HBASE-8174: -- So this becomes a backport of HBASE-8161? That patch is nice and simple. +1 > Allow each table to customize its own flush blocking file count > "hbase.hstore.blockingStoreFiles" > - > > Key: HBASE-8174 > URL: https://issues.apache.org/jira/browse/HBASE-8174 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.5 >Reporter: clockfly >Assignee: clockfly >Priority: Minor > Fix For: 0.94.7 > > Attachments: HBASE-8174.patch.0.94.v1 > > > Currently, the blocking file count "hbase.hstore.blockingStoreFiles" is > configured at region server level. > We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8161) setting blocking file count on table level doesn't work
[ https://issues.apache.org/jira/browse/HBASE-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8161: - Fix Version/s: 0.98.0 0.95.0 > setting blocking file count on table level doesn't work > --- > > Key: HBASE-8161 > URL: https://issues.apache.org/jira/browse/HBASE-8161 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.95.0, 0.98.0 > > Attachments: HBASE-8161-v0-hmm.patch, HBASE-8161-v0.patch > > > MemStoreFlusher always uses server-wide config... it should ask the store > correct question instead -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7579) HTableDescriptor equals method fails if results are returned in a different order
[ https://issues.apache.org/jira/browse/HBASE-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617979#comment-13617979 ] Lars Hofhansl commented on HBASE-7579: -- I'm thinking that in 0.94 we should just add the tests (which should not fail even in the existing code) and do the rest of the change in 0.95 and 0.98. > HTableDescriptor equals method fails if results are returned in a different > order > - > > Key: HBASE-7579 > URL: https://issues.apache.org/jira/browse/HBASE-7579 > Project: HBase > Issue Type: Bug > Components: Admin >Affects Versions: 0.95.0, 0.94.6 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman >Priority: Minor > Fix For: 0.95.0, 0.94.7 > > Attachments: HBASE-7579-v1.patch, HBASE-7579-v2.patch, > HBASE-7579-v3.patch, HBASE-7579-v4.patch > > > HTableDescriptor's compareTo function compares a set of HColumnDescriptors > against another set of HColumnDescriptors. It iterates through both, relying > on the fact that they will be in the same order. > In my testing, I may have seen this issue come up, so I decided to fix it. > It's a straightforward fix. I convert the sets into a hashset for O(1) > lookups (at least in theory), then I check that all items in the first set > are found in the second. > Since the sizes are the same, we know that if all elements showed up in the > second set, then they must be equal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-8222. -- Resolution: Fixed Hadoop Flags: Reviewed > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8188) Avoid unnecessary row compare in StoreScanner
[ https://issues.apache.org/jira/browse/HBASE-8188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8188: - Attachment: 8188-0.96.txt One last time. > Avoid unnecessary row compare in StoreScanner > - > > Key: HBASE-8188 > URL: https://issues.apache.org/jira/browse/HBASE-8188 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 8188-0.94.txt, 8188-0.96.txt, 8188-0.96.txt, > 8188-0.96.txt, 8188-0.96.txt > > > When scanner batching disable (which is the default) a row compare in > StoreScanner avoided, because only entire rows of data are requested. > This provides a slight performance gain, especially for tall tables (and when > all data is in the cache) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617976#comment-13617976
]
Jesse Yates commented on HBASE-7817:
This should be dropped...
{code}
+echo "SERVER_GC_OPTS:$SERVER_GC_OPTS"
+echo "CLIENT_GC_OPTS:$CLIENT_GC_OPTS"
+
{code}
but I can do that on commit, +1 ... I'd like to commit on monday, if there are
no objections.
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.4
>Reporter: Ian Varley
>Priority: Minor
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: HBASE-7817.patch
>
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617972#comment-13617972 ] Hudson commented on HBASE-8164: --- Integrated in hbase-0.95 #114 (See [https://builds.apache.org/job/hbase-0.95/114/]) HBASE-8164 TestTableLockManager fails intermittently in trunk builds, addendum (Revision 1462669) Result = FAILURE tedyu : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestTableLockManager.java > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617973#comment-13617973 ] Hudson commented on HBASE-8222: --- Integrated in hbase-0.95 #114 (See [https://builds.apache.org/job/hbase-0.95/114/]) HBASE-8222 User class should implement equals() and hashCode() (Revision 1462673) Result = FAILURE tedyu : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUser.java > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617962#comment-13617962 ] Ted Yu commented on HBASE-8158: --- +1 from me as well. > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-trunk-parity.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617958#comment-13617958 ] Ted Yu commented on HBASE-7904: --- Integrated patch v9 to trunk. This would use 2.0.4-SNAPSHOT for build on hadoop 2. Let's see what test failures we get on the next HBase-TRUNK-on-Hadoop-2.0.0 build. > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, > 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, > 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, > 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, > hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617956#comment-13617956 ] Ted Yu commented on HBASE-7667: --- bq. 5) Get rid of L0? Can we do this frist ? > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: Stripe compaction perf evaluation.pdf, Stripe compaction > perf evaluation.pdf, Stripe compactions.pdf, Stripe compactions.pdf, Stripe > compactions.pdf > > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[
https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617953#comment-13617953
]
Enis Soztutar commented on HBASE-8224:
--
In one of the comments here:
http://maven.40175.n5.nabble.com/How-to-deploy-with-classifier-td5523009.html ,
they basically say that one module should correspond to one artifact, and a
bunch of discussions around the jdk14, jdk15 classifiers.
How about creating one empty module per hbase module, and put the hadoop2 poms
there.
{code}
hbase-client
hbase-hadoop-2-support/hbase-client-hadoop2
hbase-server
hbase-hadoop-2-support/hbase-server-hadoop2
{code}
Then we do not even have to change the version, or have a profile for Hadoop2.
All will be build. Wdyt boss. Overkill?
> Add '-hadoop1' or '-hadoop2' to our version string
> --
>
> Key: HBASE-8224
> URL: https://issues.apache.org/jira/browse/HBASE-8224
> Project: HBase
> Issue Type: Task
>Reporter: stack
> Attachments: 8224-adding.classifiers.txt
>
>
> So we can publish both the hadoop1 and the hadoop2 jars to a maven
> repository, and so we can publish two packages, one for hadoop1 and one for
> hadoop2, given how maven works, our only alternative (to the best of my
> knowledge and after consulting others) is by amending the version string to
> include hadoop1 or hadoop2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[
https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617952#comment-13617952
]
Hadoop QA commented on HBASE-8158:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576203/8158-trunk-parity.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5062//console
This message is automatically generated.
> Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested
> inside another MR job"
> --
>
> Key: HBASE-8158
> URL: https://issues.apache.org/jira/browse/HBASE-8158
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 0.94.7
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 8158-port-jarfinder-0.94.patch,
> 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch,
> 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch,
> 8158-trunk-parity.patch
>
>
> Assess the stability of the JarFinder util class from Hadoop-2, as a
> candidate for backporting to 0.94.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617951#comment-13617951 ] Sergey Shelukhin commented on HBASE-7667: - I did a c1.xlarge test (with default, 3, 10 and 25 stripes, 2 times each). The results for different stripe configurations are very consistent across both runs. Compared to m1.large test the positive effect of increasing number of stripes on write speed is less. For this load, sweet spot appears to be around 10-12 stripes based on two tests. 3 stripes have large compactions similar to default (well, not as large); 25 stripes does too many small compactions, so select-compact loop cannot keep up with the number of files produced - on "Iteration 2" test described in the doc at least some stripes in 25-stripe case always have 6-8 small files (as they get compacted other stripes get more files). This appears to be the limiting factor on increasing the number of stripes. I think the main point is that, for count scheme, there's perf parity (writes are generally slightly slower, reads slightly faster), despite existing and fixable write amplification; and there's reduction of variability, which was the goal. I will try to devise a more realistic read workload, but I don't think it should change much given above. For sequential data, with size-based stripe scheme there's reduction in compactions, as expected, despite even L0. Next steps: 1) On existing data I want to correlate read/write perf with compactions. It is interesting that stripe scheme has slower writes in general, as Jimmy has noted - it touches read path but not anything at all on write path, so it is probably I/O related, or stresses some interaction between existing write and compaction paths. 2) Run tests for more realistic read workloads (and parallel read/writes), by not using LoadTestTool? Optional-ish. 3) Clean up integration test patch in HBASE-8000. 4) Review and commit? :) 5) Get rid of L0? > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: Stripe compaction perf evaluation.pdf, Stripe compaction > perf evaluation.pdf, Stripe compactions.pdf, Stripe compactions.pdf, Stripe > compactions.pdf > > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalanci
[jira] [Updated] (HBASE-8176) Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94
[
https://issues.apache.org/jira/browse/HBASE-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
clockfly updated HBASE-8176:
Release Note:
With HBASE-8176("Dynamic Schema Configurations"), we can define table/column
family specific configuration by HColumnDescriptor.setValue() or
HTableDescriptor.setValue(). We can also do this easily in hbase shell.
Change the table-scope by set attribute CONFIG like this:
alter 'test', METHOD => 'table_att', CONFIG => {'hbase.hstore.compaction.min'
=> '5'}
Change the column family config by set attribute CONFIG like this:
alter 'test', NAME => 'f', CONFIG => {'hbase.hstore.compaction.min' => '5'}
was:
With HBASE-8176("Dynamic Schema Configurations"), we can define table/column
family specific configuration by HColumnDescriptor.setValue() or
HTableDescriptor.setValue(). We can also do this easily in hbase shell.
Change the table-scope by set attribute CONFIG like this:
alter 'test', METHOD => 'table_att', CONFIG => {'hbase.hstore.compaction.min'
=> '5'}
Change the column family config by set attribute CONFIG like this:
alter 'test', NAME => 'f', CONFIG => {'hbase.hstore.compaction.min' => a'5'}
> Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94
> ---
>
> Key: HBASE-8176
> URL: https://issues.apache.org/jira/browse/HBASE-8176
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.5
>Reporter: clockfly
>Assignee: clockfly
>Priority: Minor
> Fix For: 0.94.7
>
> Attachments: hbase-8176.patch, HBASE-8176.patchv2,
> hbase-8176-release-notes.patch
>
>
> With HBASE-5335, we can support per-table configuration and per-family
> configurations.
> We can use it to customize the compaction on table/family basis, customize
> the flush, and etc..
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617935#comment-13617935
]
Hadoop QA commented on HBASE-7817:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576201/HBASE-7817.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5061//console
This message is automatically generated.
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.4
>Reporter: Ian Varley
>Priority: Minor
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: HBASE-7817.patch
>
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, p
[jira] [Commented] (HBASE-8211) Support for NN HA for 0.94
[ https://issues.apache.org/jira/browse/HBASE-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617931#comment-13617931 ] Fengdong Yu commented on HBASE-8211: Himanshu, I also reviewed the patch, but this just added reties, not support HDFS HA really. Since during Namenode failover, FileSystem URI will be changed, but we always try using the old "fs", I don't think this works well. Another, for HDFS HA, fs.defaultFS can be configured as "hdfs://mycluster, does this patch support "hbase.rootdir" configured "hdfs://mycluster/hbase ? > Support for NN HA for 0.94 > -- > > Key: HBASE-8211 > URL: https://issues.apache.org/jira/browse/HBASE-8211 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 0.94.6 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.94.7 > > Attachments: HBase-8211-v1.patch > > > HBase-8156 is for adding support for retrying non-idempotent operations. This > is useful in case NN is suffering from n/w hiccups, etc. This jira is to add > similar support for 0.94.x branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8208) Data could not be replicated to slaves when deferredLogSync is enabled
[ https://issues.apache.org/jira/browse/HBASE-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617925#comment-13617925 ] Jeffrey Zhong commented on HBASE-8208: -- [~lhofhansl] The newly added sync will make "deferredSync" cycle(time interval) logically shorter. Let's say we defer sync every 1 sec before the fix. With the fix, we sync either 1 sec or for each flush. Basically we have more syncs. So each sync will have less data to sync than before. If the "sync storm" you're referring to the amount of data, it will make things better. If you're referring to the frequency of sync, it won't add many extra syncs because the newly added sync is per flush and only sync when there is something to flush. In non-deferred sync case, we have much more syncs. IMHO, it should not have much impact. For this change, I with we have a good stressing test env so we can quantify the impact. To be safe, we can put it in trunk firstly and backport it to 0.94 once we're confident. Thanks, -Jeffrey > Data could not be replicated to slaves when deferredLogSync is enabled > -- > > Key: HBASE-8208 > URL: https://issues.apache.org/jira/browse/HBASE-8208 > Project: HBase > Issue Type: Bug >Affects Versions: 0.95.0, 0.98.0, 0.94.6 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: hbase-8208.patch, hbase-8208-v1.patch, > hbase-8208_v2.patch > > > This is a subtle issue. When deferredLogSync is enabled, there are chances we > could flush data before syncing all HLog entries. Assuming we just flush the > internal cache and the server dies with some unsynced hlog entries. > Data is not lost at the source cluster while replication is based on WAL > files and some changes we flushed at the source won't be replicated the slave > clusters. > Although enabling deferredLogSync with tolerances of data loss, it breaks the > replication assumption that whatever persisted in the source should be > replicated to its slave clusters. > In short, the slave cluster could end up with double losses: the data loss in > the source and some data stored in source cluster may not be replicated to > slaves either. > The fix of the issue isn't hard. Basically we can invoke sync during each > flush when replication is enabled for a region server. Since sync returns > immediately when nothing to sync so there should be no performance impact. > Please let me know what you think! > Thanks, > -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions
[
https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617917#comment-13617917
]
Hadoop QA commented on HBASE-8226:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576188/8226.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.master.TestTableLockManager
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5058//console
This message is automatically generated.
> HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts
> "too many" regions
>
>
> Key: HBASE-8226
> URL: https://issues.apache.org/jira/browse/HBASE-8226
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.95.0, 0.96.0, 0.94.7
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Attachments: 8226-branch-0.94.patch, 8226.patch, 8226.patch
>
>
> HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region
> it is counting belongs to the table created by the test and will not return
> if it accidentally counts "too many" regions, for example the regions of the
> ACL table when security is enabled.
> Made an attempt at fixing this as part of HBASE-8209 but broke
> TestMasterTransitions. Try again here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617915#comment-13617915
]
Hadoop QA commented on HBASE-7970:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576187/HBASE-7970-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5059//console
This message is automatically generated.
> Improve file descriptor usage: currently, there are two file descriptors per
> storefile
> --
>
> Key: HBASE-7970
> URL: https://issues.apache.org/jira/browse/HBASE-7970
> Project: HBase
> Issue Type: Sub-task
>Reporter: Himanshu Vashishtha
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7970-v0.patch
>
>
> This is because there are two open calls in the HFile: one with checksum and
> another for without checksum support in v2:
> see the method in HFile:createReaderWithEncoding()
> {code}
> FSDataInputStream fsdis = fs.open(path);
> FSDataInputStream fsdisNoFsChecksum = fsdis;
> // If the fs is not an instance of HFileSystem, then create an
> // instance of HFileSystem that wraps over the specified fs.
> // In this case, we will not be able to avoid checksumming inside
> // the filesystem.
> if (!(fs instanceof HFileSystem)) {
> hfs = new HFileSystem(fs);
> } else {
> hfs = (HFileSystem)fs;
> // open a stream to read data without checksum verification in
> // the filesystem
> fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7842) Add compaction policy that explores more storefile groups
[ https://issues.apache.org/jira/browse/HBASE-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617913#comment-13617913 ] Sergey Shelukhin commented on HBASE-7842: - Yeah, maybe this one should be default and the old one should be Old > Add compaction policy that explores more storefile groups > - > > Key: HBASE-7842 > URL: https://issues.apache.org/jira/browse/HBASE-7842 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-7842-0.patch, HBASE-7842-2.patch, > HBASE-7842-3.patch, HBASE-7842-4.patch, HBASE-7842-5.patch > > > Some workloads that are not as stable can have compactions that are too large > or too small using the current storefile selection algorithm. > Currently: > * Find the first file that Size(fi) <= Sum(0, i-1, FileSize(fx)) > * Ensure that there are the min number of files (if there aren't then bail > out) > * If there are too many files keep the larger ones. > I would propose something like: > * Find all sets of storefiles where every file satisfies > ** FileSize(fi) <= Sum(0, i-1, FileSize(fx)) > ** Num files in set =< max > ** Num Files in set >= min > * Then pick the set of files that maximizes ((# storefiles in set) / > Sum(FileSize(fx))) > The thinking is that the above algorithm is pretty easy reason about, all > files satisfy the ratio, and should rewrite the least amount of data to get > the biggest impact in seeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8187) trunk/0.95 tarball packaging
[ https://issues.apache.org/jira/browse/HBASE-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617908#comment-13617908 ] Hudson commented on HBASE-8187: --- Integrated in HBase-TRUNK #4000 (See [https://builds.apache.org/job/HBase-TRUNK/4000/]) HBASE-8187 trunk/0.95 tarball packaging (Revision 1462606) Result = FAILURE stack : Files : * /hbase/trunk/hbase-assembly * /hbase/trunk/hbase-assembly/pom.xml * /hbase/trunk/hbase-assembly/src * /hbase/trunk/hbase-assembly/src/assembly * /hbase/trunk/hbase-assembly/src/assembly/components.xml * /hbase/trunk/hbase-assembly/src/assembly/hadoop-one-compat.xml * /hbase/trunk/hbase-assembly/src/assembly/hadoop-two-compat.xml * /hbase/trunk/hbase-assembly/src/assembly/src.xml * /hbase/trunk/hbase-assembly/src/docbkx * /hbase/trunk/hbase-assembly/src/docbkx/book.xml * /hbase/trunk/hbase-assembly/src/docbkx/case_studies.xml * /hbase/trunk/hbase-assembly/src/docbkx/community.xml * /hbase/trunk/hbase-assembly/src/docbkx/configuration.xml * /hbase/trunk/hbase-assembly/src/docbkx/customization.xsl * /hbase/trunk/hbase-assembly/src/docbkx/developer.xml * /hbase/trunk/hbase-assembly/src/docbkx/external_apis.xml * /hbase/trunk/hbase-assembly/src/docbkx/getting_started.xml * /hbase/trunk/hbase-assembly/src/docbkx/ops_mgt.xml * /hbase/trunk/hbase-assembly/src/docbkx/performance.xml * /hbase/trunk/hbase-assembly/src/docbkx/preface.xml * /hbase/trunk/hbase-assembly/src/docbkx/rpc.xml * /hbase/trunk/hbase-assembly/src/docbkx/security.xml * /hbase/trunk/hbase-assembly/src/docbkx/shell.xml * /hbase/trunk/hbase-assembly/src/docbkx/troubleshooting.xml * /hbase/trunk/hbase-assembly/src/docbkx/upgrading.xml * /hbase/trunk/hbase-assembly/src/docbkx/zookeeper.xml * /hbase/trunk/hbase-assembly/src/site * /hbase/trunk/hbase-assembly/src/site/resources * /hbase/trunk/hbase-assembly/src/site/resources/css * /hbase/trunk/hbase-assembly/src/site/resources/css/freebsd_docbook.css * /hbase/trunk/hbase-assembly/src/site/resources/css/site.css * /hbase/trunk/hbase-assembly/src/site/resources/doap_Hbase.rdf * /hbase/trunk/hbase-assembly/src/site/resources/images * /hbase/trunk/hbase-assembly/src/site/resources/images/big_h_logo.svg * /hbase/trunk/hbase-assembly/src/site/resources/images/hbase_logo.svg * /hbase/trunk/hbase-assembly/src/site/site.vm * /hbase/trunk/hbase-assembly/src/site/site.xml * /hbase/trunk/hbase-assembly/src/site/xdoc * /hbase/trunk/hbase-assembly/src/site/xdoc/acid-semantics.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/bulk-loads.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/cygwin.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/index.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/metrics.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/old_news.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/pseudo-distributed.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/replication.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/resources.xml * /hbase/trunk/hbase-assembly/src/site/xdoc/sponsors.xml * /hbase/trunk/hbase-assembly/src/xslt * /hbase/trunk/hbase-assembly/src/xslt/configuration_to_docbook_section.xsl * /hbase/trunk/hbase-client/pom.xml * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-examples/pom.xml * /hbase/trunk/hbase-hadoop-compat/pom.xml * /hbase/trunk/hbase-hadoop1-compat/pom.xml * /hbase/trunk/hbase-hadoop2-compat/pom.xml * /hbase/trunk/hbase-it/pom.xml * /hbase/trunk/hbase-prefix-tree/pom.xml * /hbase/trunk/hbase-protocol/pom.xml * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/pom.xml * /hbase/trunk/src > trunk/0.95 tarball packaging > > > Key: HBASE-8187 > URL: https://issues.apache.org/jira/browse/HBASE-8187 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Fix For: 0.95.0 > > Attachments: 8187.txt, 8187v2.txt, 8187v3.txt, > 8187v5.095.addendum.txt, 8187v5.095.txt, 8187v5.txt > > > Packaging needs work now we have maven multi-moduled. Sounds like folks want > a package for hadoop1 and hadoop2. Also want source package and maybe even a > package that includes test jars so can run integration tests. > Let this be umbrella issue for package fixes. Will hang smaller issues off > this one as we figure them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8207) Replication could have data loss when machine name contains hyphen "-"
[
https://issues.apache.org/jira/browse/HBASE-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617907#comment-13617907
]
Hudson commented on HBASE-8207:
---
Integrated in HBase-TRUNK #4000 (See
[https://builds.apache.org/job/HBase-TRUNK/4000/])
HBASE-8207 Addendum fixes the typo in extractDeadServersFromZNodeString()
(Revision 1462552)
HBASE-8207 Replication could have data loss when machine name contains hyphen
"-" (Jeffrey) (Revision 1462515)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
tedyu :
Files :
*
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
> Replication could have data loss when machine name contains hyphen "-"
> --
>
> Key: HBASE-8207
> URL: https://issues.apache.org/jira/browse/HBASE-8207
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 8207-trunk-addendum.txt, 8207_v3.patch, failed.txt,
> hbase-8207-0.94-v1.patch, hbase-8207.patch, hbase-8207_v1.patch,
> hbase-8207_v2.patch, hbase-8207_v2.patch, HBASE-8212-94.patch
>
>
> In the recent test case TestReplication* failures, I'm finally able to find
> the cause(or one of causes) for its intermittent failures.
> When a machine name contains "-", it breaks the function
> ReplicationSource.checkIfQueueRecovered. It causes the following issue:
> deadRegionServers list is way off so that replication doesn't wait for log
> splitting finish for a wal file and move on to the next one(data loss)
> You can see that replication use those weird paths constructed from
> deadRegionServers to check a file existence
> {code}
> 2013-03-26 21:26:51,385 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,386 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,387 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,389 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,391 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,394 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,396 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/0/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,398 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1
[jira] [Commented] (HBASE-8209) Improve LoadTest extensibility
[ https://issues.apache.org/jira/browse/HBASE-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617906#comment-13617906 ] Hudson commented on HBASE-8209: --- Integrated in HBase-TRUNK #4000 (See [https://builds.apache.org/job/HBase-TRUNK/4000/]) Amend HBASE-8209. Revert changes to HBaseTestingUtility#waitUntilAllRegionsAssigned (Revision 1462639) HBASE-8209. Improve LoadTest extensibility (Revision 1462611) Result = FAILURE apurtell : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java apurtell : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > Improve LoadTest extensibility > -- > > Key: HBASE-8209 > URL: https://issues.apache.org/jira/browse/HBASE-8209 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.95.0, 0.96.0, 0.94.7 > > Attachments: addendum-0.94.patch, addendum-trunk.patch, > HBASE-8209-0.94.patch, HBASE-8209-0.94-v2.patch, HBASE-8209-0.94-v3.patch, > HBASE-8209-trunk.patch, HBASE-8209-trunk-v2.patch, HBASE-8209-trunk-v3.patch > > > This patch rolls up related changes: > - Allow classes extending TestMiniClusterLoadSequential to override how > reader and writer threads are set up, including extending the reader and > writers themselves. > - Make it so classes extending MultiThreadedWriter can cleanly override how > Puts are constructed. > - Provide an option for passing in a custom table descriptor to > HBaseTestingUtility#createPreSplitLoadTestTable. > > - HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the > region it is counting belongs to the table created by the test and will not > return if it accidentally counts "too many" regions, for example the regions > of the ACL table when security is enabled. (Some class based on > TestMiniClusterLoadSequential may want to enable security.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8208) Data could not be replicated to slaves when deferredLogSync is enabled
[ https://issues.apache.org/jira/browse/HBASE-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617904#comment-13617904 ] Lars Hofhansl commented on HBASE-8208: -- Will we get a "sync storm" when we have a lot of column families? > Data could not be replicated to slaves when deferredLogSync is enabled > -- > > Key: HBASE-8208 > URL: https://issues.apache.org/jira/browse/HBASE-8208 > Project: HBase > Issue Type: Bug >Affects Versions: 0.95.0, 0.98.0, 0.94.6 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: hbase-8208.patch, hbase-8208-v1.patch, > hbase-8208_v2.patch > > > This is a subtle issue. When deferredLogSync is enabled, there are chances we > could flush data before syncing all HLog entries. Assuming we just flush the > internal cache and the server dies with some unsynced hlog entries. > Data is not lost at the source cluster while replication is based on WAL > files and some changes we flushed at the source won't be replicated the slave > clusters. > Although enabling deferredLogSync with tolerances of data loss, it breaks the > replication assumption that whatever persisted in the source should be > replicated to its slave clusters. > In short, the slave cluster could end up with double losses: the data loss in > the source and some data stored in source cluster may not be replicated to > slaves either. > The fix of the issue isn't hard. Basically we can invoke sync during each > flush when replication is enabled for a region server. Since sync returns > immediately when nothing to sync so there should be no performance impact. > Please let me know what you think! > Thanks, > -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8211) Support for NN HA for 0.94
[ https://issues.apache.org/jira/browse/HBASE-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617903#comment-13617903 ] Lars Hofhansl commented on HBASE-8211: -- Big but straightforward change. I wish HDFS would handle this stuff for us. LGTM. > Support for NN HA for 0.94 > -- > > Key: HBASE-8211 > URL: https://issues.apache.org/jira/browse/HBASE-8211 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 0.94.6 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.94.7 > > Attachments: HBase-8211-v1.patch > > > HBase-8156 is for adding support for retrying non-idempotent operations. This > is useful in case NN is suffering from n/w hiccups, etc. This jira is to add > similar support for 0.94.x branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8158: Attachment: 8158-trunk-parity.patch Here's a patch for trunk that brings it into alignment with this new logic for 0.94. > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-trunk-parity.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8156) Support for Namenode HA for non-idempotent operations
[ https://issues.apache.org/jira/browse/HBASE-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-8156: - Fix Version/s: 0.98.0 > Support for Namenode HA for non-idempotent operations > - > > Key: HBASE-8156 > URL: https://issues.apache.org/jira/browse/HBASE-8156 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Affects Versions: 0.95.0 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Fix For: 0.95.0, 0.98.0 > > Attachments: HBase-8156-trunk-v1.patch, HBase-8156-trunk-v2.patch, > HBase-8156-trunk-v3.patch, HBase-8156-trunk-v4.patch, > HBase-8156-trunk-v4.patch, HBase-8156-trunk-v4.patch > > > In hadoop 2 HA, non-idempotent operations are not retried at the hdfs side. > This is by design as retrying a non-idempotent operation might not be a good > design choice for some use case. > HBase needs to handle the retries for such operations at its end. > With HBase-7806, there is already some work going on for file system > abstractions. There, HReginFileSystem sits as an abstraction between region > and FS. This jira is a move in the same direction, where it adds retry > functionality for non-idempotent calls such as create, rename and delete in > HRegionFileSystem class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617898#comment-13617898 ] Lars Hofhansl commented on HBASE-8222: -- Posthumous "yes please". > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617897#comment-13617897 ] Enis Soztutar commented on HBASE-8158: -- This looks good to go, and Nick says that it fixes the IT test. The exact logic in the patch seems to be now: (1) Try existing findContainingJar. (2) Try to locate JarFinder in the classpath, if found use it. (3) Fall back to backported JarFinder. There is a reordering of steps 1 and 2 in the patch compared to current code, but it should not make a difference. Any objections? [~lhofhansl] do you want some time to take a look? > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[ https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617895#comment-13617895 ] stack commented on HBASE-8224: -- Or, what am I going to do? If version is 0.97.0-hadoop1-SNAPSHOT and on command-line you specify -Dhadoop.profile=2.0? You'll get a jar that has -hadoop1 suffix though it was built against hadoop2 (I tried having the hadoop.profile property included in pom.version but it just says not allowed and fails) > Add '-hadoop1' or '-hadoop2' to our version string > -- > > Key: HBASE-8224 > URL: https://issues.apache.org/jira/browse/HBASE-8224 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: 8224-adding.classifiers.txt > > > So we can publish both the hadoop1 and the hadoop2 jars to a maven > repository, and so we can publish two packages, one for hadoop1 and one for > hadoop2, given how maven works, our only alternative (to the best of my > knowledge and after consulting others) is by amending the version string to > include hadoop1 or hadoop2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[
https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8224:
-
Attachment: 8224-adding.classifiers.txt
I tried adding classifiers. Was fine till I got to the assembly step:
{code}
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.4:single (default-cli) on
project hbase-assembly: Failed to create assembly: Error adding file
'org.apache.hbase:hbase-common:jar:0.97.0-SNAPSHOT' to archive:
/Users/stack/checkouts/hbase/hbase-common/target/classes isn't a file. -> [Help
1]
{code}
I couldn't get over the above. There is no vocabulary around moduleSet and
dependencySet to deal w/ classifiers; they seem blind unless I try and call
out what to include individually (failed trying to do this too).
Giving up on this avenue. Going to add to our version.
> Add '-hadoop1' or '-hadoop2' to our version string
> --
>
> Key: HBASE-8224
> URL: https://issues.apache.org/jira/browse/HBASE-8224
> Project: HBase
> Issue Type: Task
>Reporter: stack
> Attachments: 8224-adding.classifiers.txt
>
>
> So we can publish both the hadoop1 and the hadoop2 jars to a maven
> repository, and so we can publish two packages, one for hadoop1 and one for
> hadoop2, given how maven works, our only alternative (to the best of my
> knowledge and after consulting others) is by amending the version string to
> include hadoop1 or hadoop2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vasu Mariyala updated HBASE-7817:
-
Affects Version/s: 0.94.4
Status: Patch Available (was: Open)
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.4
>Reporter: Ian Varley
>Priority: Minor
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: HBASE-7817.patch
>
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vasu Mariyala updated HBASE-7817:
-
Attachment: HBASE-7817.patch
Background of the issue:
Initializing the common configuration and setting up the environment variables
is done by hbase-config.sh. This file in turn sources hbase-env.sh to set up
the environment variables. For few environment variables which contain multiple
configurations, new content is appended to the existing content.
Example:
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $HBASE_JMX_BASE
-Dcom.sun.management.jmxremote.port=10101"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
Since most of the scripts like hbase-daemons.sh, hbase-daemon.sh can be run
independently by the users, all of these scripts source the hbase-config.sh. As
a result, hbase-config.sh and there by hbase-env.sh are invoked multiple times
in a flow. Due to this, few environment variables like HBASE_MASTER_OPTS
contain duplicate data. In the scenarios like the jdwp debug options, the
duplicate content would not let hbase start successfully.
In addition to the above, there is a cyclic dependency between hbase-env.sh and
hbase-daemon.sh
a) HBase-env.sh can export HBASE_USE_GC_LOGFILE
export HBASE_USE_GC_LOGFILE=true
b) HBase-Daemon.sh sets HBASE_GC_OPTS
if [ "$HBASE_USE_GC_LOGFILE" = "true" ]; then
export HBASE_GC_OPTS=" -Xloggc:${loggc}"
fi
c) Back in HBase-env.sh, we have the SERVER_GC_OPTS and CLIENT_GC_OPTS depend
on the HBASE_GC_OPTS set in HBase-daemon.sh as below
export SERVER_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
$HBASE_GC_OPTS"
export CLIENT_GC_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
$HBASE_GC_OPTS"
The patch fixes the issue mentioned in this jia and also removes the cyclic
dependency between these scripts.
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Reporter: Ian Varley
>Priority: Minor
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: HBASE-7817.patch
>
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7615) Add metrics for snapshots
[
https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617890#comment-13617890
]
Ted Yu commented on HBASE-7615:
---
nit:
{code}
+<%= stats.isSnapshotCorrupted() ? "CORRUPTED" : "ok" %>
{code}
Consider showing the "CORRUPTED" text in a different color.
+1 on patch, if tests pass.
> Add metrics for snapshots
> -
>
> Key: HBASE-7615
> URL: https://issues.apache.org/jira/browse/HBASE-7615
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
> Attachments: HBASE-7615-0.94.patch, HBASE-7615-v0.patch,
> HBASE-7615-v0-ui-corrupted.png, HBASE-7615-v0-ui.png, HBASE-7615-v1.patch,
> HBASE-7615-v2.patch
>
>
> Metrics should be added for snapshot.
> From Matteo: Output that we have in SnapshotInfo should be covered:
> "%d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source
> table)"
> Jesse mentioned snaphot counts, average time to completion.
> I think we should have counts for successful / failed snapshots.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[
https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617891#comment-13617891
]
Hadoop QA commented on HBASE-8158:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576200/8158-port-jarfinder-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5060//console
This message is automatically generated.
> Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested
> inside another MR job"
> --
>
> Key: HBASE-8158
> URL: https://issues.apache.org/jira/browse/HBASE-8158
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 0.94.7
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
> Attachments: 8158-port-jarfinder-0.94.patch,
> 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch,
> 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch
>
>
> Assess the stability of the JarFinder util class from Hadoop-2, as a
> candidate for backporting to 0.94.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7938) Add integration test for ImportTsv/LoadIncrementalHFiles workflow
[ https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7938: Release Note: The new IntegrationTestImportTsv is written in the style of the recently added IntegrationTestLoadAndVerify. It is designed to be run from maven or directly against a distributed cluster. It shares core test logic with TestImportTsv. To run it with maven, first refresh your install jars, and the run the test: $ mvn clean package install -DskipTests $ cd hbase-it $ mvn failsafe:integration-test -Dit.test=IntegrationTestImportTsv NOTE: Running from maven fails because HFileOutputFormat is creating the partition file on localFs, not HDFS. Or run it stand-alone: $ HBASE_CLASSPATH="hbase-it/target/hbase-it--tests.jar" \ ./bin/hbase org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv When run stand-alone, it supports all the usual Hadoop Tool arguments, particularly -conf. ImportTsv was refactored to make code sharing easier. It is upgraded from MediumTest to LargeTest because it take 180+ seconds to run on my machine. Tests for the TsvParser are separated into their own SmallTest. > Add integration test for ImportTsv/LoadIncrementalHFiles workflow > - > > Key: HBASE-7938 > URL: https://issues.apache.org/jira/browse/HBASE-7938 > Project: HBase > Issue Type: Sub-task > Components: mapreduce >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Fix For: 0.95.0, 0.98.0 > > Attachments: > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch, > 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch > > > We have existing unit tests for smoke-testing the packaged MR jobs, however > they do not create a runtime environment that is true to running on a real MR > cluster. This is particularly true in regard to classpaths (HBASE-7934) but > also other static state (HBASE-4802). An integration test that can be pointed > to run on a pseudo-distributed Hadoop deployed on localhost would find these > kinds of problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8208) Data could not be replicated to slaves when deferredLogSync is enabled
[ https://issues.apache.org/jira/browse/HBASE-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617883#comment-13617883 ] Ted Yu commented on HBASE-8208: --- Patch v2 looks good to me. > Data could not be replicated to slaves when deferredLogSync is enabled > -- > > Key: HBASE-8208 > URL: https://issues.apache.org/jira/browse/HBASE-8208 > Project: HBase > Issue Type: Bug >Affects Versions: 0.95.0, 0.98.0, 0.94.6 >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: hbase-8208.patch, hbase-8208-v1.patch, > hbase-8208_v2.patch > > > This is a subtle issue. When deferredLogSync is enabled, there are chances we > could flush data before syncing all HLog entries. Assuming we just flush the > internal cache and the server dies with some unsynced hlog entries. > Data is not lost at the source cluster while replication is based on WAL > files and some changes we flushed at the source won't be replicated the slave > clusters. > Although enabling deferredLogSync with tolerances of data loss, it breaks the > replication assumption that whatever persisted in the source should be > replicated to its slave clusters. > In short, the slave cluster could end up with double losses: the data loss in > the source and some data stored in source cluster may not be replicated to > slaves either. > The fix of the issue isn't hard. Basically we can invoke sync during each > flush when replication is enabled for a region server. Since sync returns > immediately when nothing to sync so there should be no performance impact. > Please let me know what you think! > Thanks, > -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8228) Investigate time taken to snapshot memstore
Amitanand Aiyer created HBASE-8228: -- Summary: Investigate time taken to snapshot memstore Key: HBASE-8228 URL: https://issues.apache.org/jira/browse/HBASE-8228 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Priority: Minor Snapshotting memstores is normally quick. But, sometimes it seems to take long. This JIRA is to track the investigation and fix to improve the outliers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8227) Investigate outliers in 0.89-fb
Amitanand Aiyer created HBASE-8227: -- Summary: Investigate outliers in 0.89-fb Key: HBASE-8227 URL: https://issues.apache.org/jira/browse/HBASE-8227 Project: HBase Issue Type: Improvement Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89-fb Umbrella task to investigate outliers in the get and put path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617881#comment-13617881 ] Ted Yu commented on HBASE-8222: --- Integrated to 0.94 as well. > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8158: Attachment: 8158-port-jarfinder-0.94.patch This patch uses reflection only for hadoop's jarfinder. > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8222: -- Attachment: 8222-v2-0.94.txt TestUser passes on 0.94: Running org.apache.hadoop.hbase.security.TestUser 2013-03-29 16:30:29.339 java[10638:1203] Unable to load realm info from SCDynamicStore Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.402 sec > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8222: -- Attachment: (was: 8222-v2.txt) > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2-0.94.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8222: -- Status: Open (was: Patch Available) > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8208) Data could not be replicated to slaves when deferredLogSync is enabled
[
https://issues.apache.org/jira/browse/HBASE-8208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617875#comment-13617875
]
Hadoop QA commented on HBASE-8208:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576181/hbase-8208_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5057//console
This message is automatically generated.
> Data could not be replicated to slaves when deferredLogSync is enabled
> --
>
> Key: HBASE-8208
> URL: https://issues.apache.org/jira/browse/HBASE-8208
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0, 0.98.0, 0.94.6
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: hbase-8208.patch, hbase-8208-v1.patch,
> hbase-8208_v2.patch
>
>
> This is a subtle issue. When deferredLogSync is enabled, there are chances we
> could flush data before syncing all HLog entries. Assuming we just flush the
> internal cache and the server dies with some unsynced hlog entries.
> Data is not lost at the source cluster while replication is based on WAL
> files and some changes we flushed at the source won't be replicated the slave
> clusters.
> Although enabling deferredLogSync with tolerances of data loss, it breaks the
> replication assumption that whatever persisted in the source should be
> replicated to its slave clusters.
> In short, the slave cluster could end up with double losses: the data loss in
> the source and some data stored in source cluster may not be replicated to
> slaves either.
> The fix of the issue isn't hard. Basically we can invoke sync during each
> flush when replication is enabled for a region server. Since sync returns
> immediately when nothing to sync so there should be no performance impact.
> Please let me know what you think!
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617870#comment-13617870 ] Ted Yu commented on HBASE-8222: --- Integrated to 0.95 and trunk. Thanks for reviews, Andy and Gary. Will integrate to 0.94 soon. > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8222) User class should implement equals() and hashCode()
[ https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8222: -- Summary: User class should implement equals() and hashCode() (was: User should implement equals() and hashCode()) > User class should implement equals() and hashCode() > --- > > Key: HBASE-8222 > URL: https://issues.apache.org/jira/browse/HBASE-8222 > Project: HBase > Issue Type: Bug > Components: security >Affects Versions: 0.94.6 >Reporter: Gary Helmling >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0, 0.98.0, 0.94.7 > > Attachments: 8222-v1.txt, 8222-v2.txt, 8222-v2.txt > > > The current implementation of the User class does not override equals() and > hashCode(), so connection reuse in HBaseClient is completely broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8222) User should implement equals() and hashCode()
[
https://issues.apache.org/jira/browse/HBASE-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617868#comment-13617868
]
Hadoop QA commented on HBASE-8222:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576164/8222-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5055//console
This message is automatically generated.
> User should implement equals() and hashCode()
> -
>
> Key: HBASE-8222
> URL: https://issues.apache.org/jira/browse/HBASE-8222
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.94.6
>Reporter: Gary Helmling
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 8222-v1.txt, 8222-v2.txt, 8222-v2.txt
>
>
> The current implementation of the User class does not override equals() and
> hashCode(), so connection reuse in HBaseClient is completely broken.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7383) create integration test for HBASE-5416 (improving scan performance for certain filters)
[ https://issues.apache.org/jira/browse/HBASE-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617864#comment-13617864 ] Hudson commented on HBASE-7383: --- Integrated in HBase-0.94 #927 (See [https://builds.apache.org/job/HBase-0.94/927/]) HBASE-8210. Backport the LoadTest portions of HBASE-7383 (Revision 1462511) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestDataGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > create integration test for HBASE-5416 (improving scan performance for > certain filters) > --- > > Key: HBASE-7383 > URL: https://issues.apache.org/jira/browse/HBASE-7383 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.95.0 > > Attachments: HBASE-7383-v0.patch, HBASE-7383-v1.patch, > HBASE-7383-v1.patch, HBASE-7383-v2.patch, HBASE-7383-v3.patch > > > HBASE-5416 is risky and needs an integration test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8210) Backport the LoadTest portions of HBASE-7383
[ https://issues.apache.org/jira/browse/HBASE-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617865#comment-13617865 ] Hudson commented on HBASE-8210: --- Integrated in HBase-0.94 #927 (See [https://builds.apache.org/job/HBase-0.94/927/]) HBASE-8210. Backport the LoadTest portions of HBASE-7383 (Revision 1462511) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestDataGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestKVGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/LoadTestTool.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedReader.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/RestartMetaTest.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > Backport the LoadTest portions of HBASE-7383 > > > Key: HBASE-8210 > URL: https://issues.apache.org/jira/browse/HBASE-8210 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.7 > > Attachments: HBASE-8210-0.94.patch, HBASE-8210-0.94-v2.patch > > > This provides a consistent foundation for applying LoadTest changes to both > 0.94 and trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617863#comment-13617863 ] Hudson commented on HBASE-5335: --- Integrated in HBase-0.94 #927 (See [https://builds.apache.org/job/HBase-0.94/927/]) HBASE-8176 Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94 (clockfly) (Revision 1462622) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/ruby/hbase.rb * /hbase/branches/0.94/src/main/ruby/hbase/admin.rb * /hbase/branches/0.94/src/main/ruby/shell/commands/alter.rb * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java > Dynamic Schema Configurations > - > > Key: HBASE-5335 > URL: https://issues.apache.org/jira/browse/HBASE-5335 > Project: HBase > Issue Type: New Feature >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg > Labels: configuration, schema > Fix For: 0.95.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--D2247.1.patch, > ASF.LICENSE.NOT.GRANTED--D2247.2.patch, > ASF.LICENSE.NOT.GRANTED--D2247.3.patch, > ASF.LICENSE.NOT.GRANTED--D2247.4.patch, > ASF.LICENSE.NOT.GRANTED--D2247.5.patch, > ASF.LICENSE.NOT.GRANTED--D2247.6.patch, > ASF.LICENSE.NOT.GRANTED--D2247.7.patch, > ASF.LICENSE.NOT.GRANTED--D2247.8.patch, HBASE-5335-trunk-2.patch, > HBASE-5335-trunk-3.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-4.patch, > HBASE-5335-trunk.patch > > > Currently, the ability for a core developer to add per-table & per-CF > configuration settings is very heavyweight. You need to add a reserved > keyword all the way up the stack & you have to support this variable > long-term if you're going to expose it explicitly to the user. This has > ended up with using Configuration.get() a lot because it is lightweight and > you can tweak settings while you're trying to understand system behavior > [since there are many config params that may never need to be tuned]. We > need to add the ability to put & read arbitrary KV settings in the HBase > schema. Combined with online schema change, this will allow us to safely > iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8176) Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617862#comment-13617862 ] Hudson commented on HBASE-8176: --- Integrated in HBase-0.94 #927 (See [https://builds.apache.org/job/HBase-0.94/927/]) HBASE-8176 Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94 (clockfly) (Revision 1462622) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/ruby/hbase.rb * /hbase/branches/0.94/src/main/ruby/hbase/admin.rb * /hbase/branches/0.94/src/main/ruby/shell/commands/alter.rb * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java > Backport HBASE-5335 "Dynamic Schema Configurations" to 0.94 > --- > > Key: HBASE-8176 > URL: https://issues.apache.org/jira/browse/HBASE-8176 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: clockfly >Assignee: clockfly >Priority: Minor > Fix For: 0.94.7 > > Attachments: hbase-8176.patch, HBASE-8176.patchv2, > hbase-8176-release-notes.patch > > > With HBASE-5335, we can support per-table configuration and per-family > configurations. > We can use it to customize the compaction on table/family basis, customize > the flush, and etc.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8207) Replication could have data loss when machine name contains hyphen "-"
[
https://issues.apache.org/jira/browse/HBASE-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617861#comment-13617861
]
Hudson commented on HBASE-8207:
---
Integrated in HBase-0.94 #927 (See
[https://builds.apache.org/job/HBase-0.94/927/])
HBASE-8207 Addendum fixes the typo in extractDeadServersFromZNodeString()
(Revision 1462554)
HBASE-8207 Replication could have data loss when machine name contains hyphen
"-" (Jeffrey) (Revision 1462518)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
tedyu :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
> Replication could have data loss when machine name contains hyphen "-"
> --
>
> Key: HBASE-8207
> URL: https://issues.apache.org/jira/browse/HBASE-8207
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 8207-trunk-addendum.txt, 8207_v3.patch, failed.txt,
> hbase-8207-0.94-v1.patch, hbase-8207.patch, hbase-8207_v1.patch,
> hbase-8207_v2.patch, hbase-8207_v2.patch, HBASE-8212-94.patch
>
>
> In the recent test case TestReplication* failures, I'm finally able to find
> the cause(or one of causes) for its intermittent failures.
> When a machine name contains "-", it breaks the function
> ReplicationSource.checkIfQueueRecovered. It causes the following issue:
> deadRegionServers list is way off so that replication doesn't wait for log
> splitting finish for a wal file and move on to the next one(data loss)
> You can see that replication use those weird paths constructed from
> deadRegionServers to check a file existence
> {code}
> 2013-03-26 21:26:51,385 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,386 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,387 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,389 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,391 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,394 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,396 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/0/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,398 INFO
> [ReplicationExecut
[jira] [Commented] (HBASE-8209) Improve LoadTest extensibility
[ https://issues.apache.org/jira/browse/HBASE-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617860#comment-13617860 ] Hudson commented on HBASE-8209: --- Integrated in HBase-0.94 #927 (See [https://builds.apache.org/job/HBase-0.94/927/]) Amend HBASE-8209. Revert changes to HBaseTestingUtility#waitUntilAllRegionsAssigned (Revision 1462641) HBASE-8209. Improve LoadTest extensibility (Revision 1462618) Result = FAILURE apurtell : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java apurtell : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestLoadTestKVGenerator.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > Improve LoadTest extensibility > -- > > Key: HBASE-8209 > URL: https://issues.apache.org/jira/browse/HBASE-8209 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.95.0, 0.96.0, 0.94.7 > > Attachments: addendum-0.94.patch, addendum-trunk.patch, > HBASE-8209-0.94.patch, HBASE-8209-0.94-v2.patch, HBASE-8209-0.94-v3.patch, > HBASE-8209-trunk.patch, HBASE-8209-trunk-v2.patch, HBASE-8209-trunk-v3.patch > > > This patch rolls up related changes: > - Allow classes extending TestMiniClusterLoadSequential to override how > reader and writer threads are set up, including extending the reader and > writers themselves. > - Make it so classes extending MultiThreadedWriter can cleanly override how > Puts are constructed. > - Provide an option for passing in a custom table descriptor to > HBaseTestingUtility#createPreSplitLoadTestTable. > > - HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the > region it is counting belongs to the table created by the test and will not > return if it accidentally counts "too many" regions, for example the regions > of the ACL table when security is enabled. (Some class based on > TestMiniClusterLoadSequential may want to enable security.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617854#comment-13617854 ] Enis Soztutar commented on HBASE-8158: -- The logic seems good. However, in case we fall back to backport JarFinder, we shouldn't be accessing it through reflection. Can you just add the import and do a direct call. > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7817:
-
Fix Version/s: 0.94.7
0.98.0
0.95.0
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Reporter: Ian Varley
>Priority: Minor
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8187) trunk/0.95 tarball packaging
[ https://issues.apache.org/jira/browse/HBASE-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617849#comment-13617849 ] Hudson commented on HBASE-8187: --- Integrated in hbase-0.95 #113 (See [https://builds.apache.org/job/hbase-0.95/113/]) HBASE-8187 trunk/0.95 tarball packaging; ADDENDUM for 0.95 (Revision 1462620) HBASE-8187 trunk/0.95 tarball packaging (Revision 1462608) Result = FAILURE stack : Files : * /hbase/branches/0.95/hbase-assembly/pom.xml * /hbase/branches/0.95/hbase-client/pom.xml * /hbase/branches/0.95/hbase-common/pom.xml * /hbase/branches/0.95/hbase-examples/pom.xml * /hbase/branches/0.95/hbase-hadoop-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop1-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop2-compat/pom.xml * /hbase/branches/0.95/hbase-it/pom.xml * /hbase/branches/0.95/hbase-prefix-tree/pom.xml * /hbase/branches/0.95/hbase-protocol/pom.xml * /hbase/branches/0.95/hbase-server/pom.xml * /hbase/branches/0.95/pom.xml stack : Files : * /hbase/branches/0.95/hbase-assembly * /hbase/branches/0.95/hbase-assembly/pom.xml * /hbase/branches/0.95/hbase-assembly/src * /hbase/branches/0.95/hbase-assembly/src/assembly * /hbase/branches/0.95/hbase-assembly/src/assembly/components.xml * /hbase/branches/0.95/hbase-assembly/src/assembly/hadoop-one-compat.xml * /hbase/branches/0.95/hbase-assembly/src/assembly/hadoop-two-compat.xml * /hbase/branches/0.95/hbase-assembly/src/assembly/src.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx * /hbase/branches/0.95/hbase-assembly/src/docbkx/book.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/case_studies.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/community.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/configuration.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/customization.xsl * /hbase/branches/0.95/hbase-assembly/src/docbkx/developer.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/external_apis.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/getting_started.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/ops_mgt.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/performance.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/preface.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/rpc.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/security.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/shell.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/troubleshooting.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/upgrading.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/zookeeper.xml * /hbase/branches/0.95/hbase-assembly/src/site * /hbase/branches/0.95/hbase-assembly/src/site/resources * /hbase/branches/0.95/hbase-assembly/src/site/resources/css * /hbase/branches/0.95/hbase-assembly/src/site/resources/css/freebsd_docbook.css * /hbase/branches/0.95/hbase-assembly/src/site/resources/css/site.css * /hbase/branches/0.95/hbase-assembly/src/site/resources/doap_Hbase.rdf * /hbase/branches/0.95/hbase-assembly/src/site/resources/images * /hbase/branches/0.95/hbase-assembly/src/site/resources/images/big_h_logo.svg * /hbase/branches/0.95/hbase-assembly/src/site/resources/images/hbase_logo.svg * /hbase/branches/0.95/hbase-assembly/src/site/site.vm * /hbase/branches/0.95/hbase-assembly/src/site/site.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/acid-semantics.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/bulk-loads.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/cygwin.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/index.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/metrics.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/old_news.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/pseudo-distributed.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/replication.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/resources.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/sponsors.xml * /hbase/branches/0.95/hbase-assembly/src/xslt * /hbase/branches/0.95/hbase-assembly/src/xslt/configuration_to_docbook_section.xsl * /hbase/branches/0.95/hbase-client/pom.xml * /hbase/branches/0.95/hbase-common/pom.xml * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-examples/pom.xml * /hbase/branches/0.95/hbase-hadoop-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop1-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop2-compat/pom.xml * /hbase/branches/0.95/hbase-it/pom.xml * /hbase/branches/0.95/hbase-prefix-tree/pom.xml * /hbase/branches/0.95/hbase-protocol/pom.xml * /hbase/branches/0.95/hbase-server/pom.xml * /hbase/branches/0.95/pom.xml * /hbase/branches/0.95/src > trunk/0.95 tarball packaging > > > Key: HBASE-8187 > URL: https://issues.apache.org/jira/browse/HBA
[jira] [Commented] (HBASE-8207) Replication could have data loss when machine name contains hyphen "-"
[
https://issues.apache.org/jira/browse/HBASE-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617848#comment-13617848
]
Hudson commented on HBASE-8207:
---
Integrated in hbase-0.95 #113 (See
[https://builds.apache.org/job/hbase-0.95/113/])
HBASE-8207 Addendum fixes the typo in extractDeadServersFromZNodeString()
(Revision 1462553)
HBASE-8207 Replication could have data loss when machine name contains hyphen
"-" (Jeffrey) (Revision 1462516)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
tedyu :
Files :
*
/hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
> Replication could have data loss when machine name contains hyphen "-"
> --
>
> Key: HBASE-8207
> URL: https://issues.apache.org/jira/browse/HBASE-8207
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 8207-trunk-addendum.txt, 8207_v3.patch, failed.txt,
> hbase-8207-0.94-v1.patch, hbase-8207.patch, hbase-8207_v1.patch,
> hbase-8207_v2.patch, hbase-8207_v2.patch, HBASE-8212-94.patch
>
>
> In the recent test case TestReplication* failures, I'm finally able to find
> the cause(or one of causes) for its intermittent failures.
> When a machine name contains "-", it breaks the function
> ReplicationSource.checkIfQueueRecovered. It causes the following issue:
> deadRegionServers list is way off so that replication doesn't wait for log
> splitting finish for a wal file and move on to the next one(data loss)
> You can see that replication use those weird paths constructed from
> deadRegionServers to check a file existence
> {code}
> 2013-03-26 21:26:51,385 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,386 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,387 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,389 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,391 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,394 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,396 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/0/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,398 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-wes
[jira] [Commented] (HBASE-8209) Improve LoadTest extensibility
[ https://issues.apache.org/jira/browse/HBASE-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617847#comment-13617847 ] Hudson commented on HBASE-8209: --- Integrated in hbase-0.95 #113 (See [https://builds.apache.org/job/hbase-0.95/113/]) Amend HBASE-8209. Revert changes to HBaseTestingUtility#waitUntilAllRegionsAssigned (Revision 1462640) HBASE-8209. Improve LoadTest extensibility (Revision 1462630) Result = FAILURE apurtell : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java apurtell : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > Improve LoadTest extensibility > -- > > Key: HBASE-8209 > URL: https://issues.apache.org/jira/browse/HBASE-8209 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.95.0, 0.96.0, 0.94.7 > > Attachments: addendum-0.94.patch, addendum-trunk.patch, > HBASE-8209-0.94.patch, HBASE-8209-0.94-v2.patch, HBASE-8209-0.94-v3.patch, > HBASE-8209-trunk.patch, HBASE-8209-trunk-v2.patch, HBASE-8209-trunk-v3.patch > > > This patch rolls up related changes: > - Allow classes extending TestMiniClusterLoadSequential to override how > reader and writer threads are set up, including extending the reader and > writers themselves. > - Make it so classes extending MultiThreadedWriter can cleanly override how > Puts are constructed. > - Provide an option for passing in a custom table descriptor to > HBaseTestingUtility#createPreSplitLoadTestTable. > > - HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the > region it is counting belongs to the table created by the test and will not > return if it accidentally counts "too many" regions, for example the regions > of the ACL table when security is enabled. (Some class based on > TestMiniClusterLoadSequential may want to enable security.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8158: Attachment: 8158-port-jarfinder-0.94.patch > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8158: Attachment: (was: 0001-HBASE-8158-backport-HBASE-8140-addDependencyJar-shou.patch) > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617844#comment-13617844 ] Ted Yu commented on HBASE-7904: --- bq. But startMiniCluster() only calls startMiniDFSCluster(), startMiniZKCluster() and startMiniHBaseCluster(), in that order. I was just saying that cluster (a MiniHBaseCluster) wouldn't get its config from UTIL.getConfiguration because the effective config is updated in UTIL.startMiniMapReduceCluster(). bq. Can open a new one if still incompatibility issues running against hadoop2. True. Another JIRA would be needed when 2.0.4-ALPHA is released. In the mean time, if we see mapreduce errors using 2.0 SNAPSHOT build, we can provide feedback upstream. > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, > 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, > 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, > 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, > hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8158) Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested inside another MR job"
[ https://issues.apache.org/jira/browse/HBASE-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-8158: Attachment: 0001-HBASE-8158-backport-HBASE-8140-addDependencyJar-shou.patch > Backport HBASE-8140 "TableMapReduceUtils#addDependencyJar fails when nested > inside another MR job" > -- > > Key: HBASE-8158 > URL: https://issues.apache.org/jira/browse/HBASE-8158 > Project: HBase > Issue Type: Bug > Components: mapreduce >Affects Versions: 0.94.7 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: 8158-port-jarfinder-0.94.patch, > 8158-port-jarfinder-0.94.patch, 8158-port-jarfinder-0.94.patch > > > Assess the stability of the JarFinder util class from Hadoop-2, as a > candidate for backporting to 0.94. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617841#comment-13617841 ] stack commented on HBASE-7904: -- I don't follow at all what you are saying above. You are not sure what fixed the problem and you present a spaghetti list of how we start up clusters in our test util to conclude "a little extra manipulation is needed...". +1 on patch. I want to put this issue out of its misery. Can open a new one if still incompatibility issues running against hadoop2. > Make mapreduce jobs pass based on 2.0.4-alpha > - > > Key: HBASE-7904 > URL: https://issues.apache.org/jira/browse/HBASE-7904 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Critical > Fix For: 0.95.0 > > Attachments: 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, > 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, > 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, > 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, > hbase-7904-v3.txt > > > 2.0.3-alpha has been released. > We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7967) implement compactor for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617842#comment-13617842 ] Jimmy Xiang commented on HBASE-7967: bq. Do you think it's worth changing now? Probably not. bq. The doc attached to this JIRA describes all that. Doesn't have pictures though Do you mean on top of that doc. The doc does have some explanation. A picture may be easier to follow. > implement compactor for stripe compactions > -- > > Key: HBASE-7967 > URL: https://issues.apache.org/jira/browse/HBASE-7967 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7967-v0.patch, HBASE-7967-v0-with-stuff.patch, > HBASE-7967-v1.patch, HBASE-7967-v1-with-7679-7680.patch, HBASE-7967-v2.patch, > HBASE-7967-v2-with-7679-7680.patch > > > Compactor needs to be implemented. See details in parent and blocking jira. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617840#comment-13617840 ] Ted Yu commented on HBASE-8164: --- Integrated to 0.95 and trunk. Thanks for the reviews, Jimmy and Enis. Still keeping this issue open - maybe there is other case I missed. > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7842) Add compaction policy that explores more storefile groups
[ https://issues.apache.org/jira/browse/HBASE-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617833#comment-13617833 ] Elliott Clark commented on HBASE-7842: -- I'd rather leave it there, just in case someone has a workload that relied on the old behavior. Perhaps it should be re-named ? > Add compaction policy that explores more storefile groups > - > > Key: HBASE-7842 > URL: https://issues.apache.org/jira/browse/HBASE-7842 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-7842-0.patch, HBASE-7842-2.patch, > HBASE-7842-3.patch, HBASE-7842-4.patch, HBASE-7842-5.patch > > > Some workloads that are not as stable can have compactions that are too large > or too small using the current storefile selection algorithm. > Currently: > * Find the first file that Size(fi) <= Sum(0, i-1, FileSize(fx)) > * Ensure that there are the min number of files (if there aren't then bail > out) > * If there are too many files keep the larger ones. > I would propose something like: > * Find all sets of storefiles where every file satisfies > ** FileSize(fi) <= Sum(0, i-1, FileSize(fx)) > ** Num files in set =< max > ** Num Files in set >= min > * Then pick the set of files that maximizes ((# storefiles in set) / > Sum(FileSize(fx))) > The thinking is that the above algorithm is pretty easy reason about, all > files satisfy the ratio, and should rewrite the least amount of data to get > the biggest impact in seeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server
[ https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617835#comment-13617835 ] Sergey Shelukhin commented on HBASE-7649: - ping? > client retry timeout doesn't need to do x2 fallback when going to different > server > -- > > Key: HBASE-7649 > URL: https://issues.apache.org/jira/browse/HBASE-7649 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7649-v0.patch, HBASE-7649-v1.patch, > HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, > HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, > HBASE-7649-v6.patch, HBASE-7649-v7.patch > > > See HBASE-7520. When we go to server A, get a bunch of failures, then finally > learn the region is on B it doesn't make sense to wait for 30 seconds before > going to B. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617832#comment-13617832 ] Enis Soztutar commented on HBASE-8164: -- bq. One of the step in modifying a table is to reassign all regions. My understanding is that the split goes to the region server when the region is closed so it is not served. Split region locks the table only after the region is confirmed to be on the region server and the split request is accepted. Yes, it is indeed what seems to be happening. In this case +1 to the addendum patch. > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7842) Add compaction policy that explores more storefile groups
[ https://issues.apache.org/jira/browse/HBASE-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617829#comment-13617829 ] Sergey Shelukhin commented on HBASE-7842: - Looks good as such; one question left is: bq. Regardless, it appears that this patch both replaces default policy and hijacks the default test, so the default ratio algorithm becomes a poor bastard child, not used and not tested Should we delete it altogether and swap it with yours in place? Do you want to just replace the default one? > Add compaction policy that explores more storefile groups > - > > Key: HBASE-7842 > URL: https://issues.apache.org/jira/browse/HBASE-7842 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-7842-0.patch, HBASE-7842-2.patch, > HBASE-7842-3.patch, HBASE-7842-4.patch, HBASE-7842-5.patch > > > Some workloads that are not as stable can have compactions that are too large > or too small using the current storefile selection algorithm. > Currently: > * Find the first file that Size(fi) <= Sum(0, i-1, FileSize(fx)) > * Ensure that there are the min number of files (if there aren't then bail > out) > * If there are too many files keep the larger ones. > I would propose something like: > * Find all sets of storefiles where every file satisfies > ** FileSize(fi) <= Sum(0, i-1, FileSize(fx)) > ** Num files in set =< max > ** Num Files in set >= min > * Then pick the set of files that maximizes ((# storefiles in set) / > Sum(FileSize(fx))) > The thinking is that the above algorithm is pretty easy reason about, all > files satisfy the ratio, and should rewrite the least amount of data to get > the biggest impact in seeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7842) Add compaction policy that explores more storefile groups
[ https://issues.apache.org/jira/browse/HBASE-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617830#comment-13617830 ] Sergey Shelukhin commented on HBASE-7842: - quote should not be cut... > Add compaction policy that explores more storefile groups > - > > Key: HBASE-7842 > URL: https://issues.apache.org/jira/browse/HBASE-7842 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-7842-0.patch, HBASE-7842-2.patch, > HBASE-7842-3.patch, HBASE-7842-4.patch, HBASE-7842-5.patch > > > Some workloads that are not as stable can have compactions that are too large > or too small using the current storefile selection algorithm. > Currently: > * Find the first file that Size(fi) <= Sum(0, i-1, FileSize(fx)) > * Ensure that there are the min number of files (if there aren't then bail > out) > * If there are too many files keep the larger ones. > I would propose something like: > * Find all sets of storefiles where every file satisfies > ** FileSize(fi) <= Sum(0, i-1, FileSize(fx)) > ** Num files in set =< max > ** Num Files in set >= min > * Then pick the set of files that maximizes ((# storefiles in set) / > Sum(FileSize(fx))) > The thinking is that the above algorithm is pretty easy reason about, all > files satisfy the ratio, and should rewrite the least amount of data to get > the biggest impact in seeks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617822#comment-13617822 ] Jimmy Xiang commented on HBASE-8164: One of the step in modifying a table is to reassign all regions. My understanding is that the split goes to the region server when the region is closed so it is not served. Split region locks the table only after the region is confirmed to be on the region server and the split request is accepted. > TestTableLockManager fails intermittently in trunk builds > - > > Key: HBASE-8164 > URL: https://issues.apache.org/jira/browse/HBASE-8164 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.95.0, 0.98.0 > > Attachments: 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt > > > In build #3979: > testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test > timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions
[ https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8226: -- Status: Open (was: Patch Available) > HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts > "too many" regions > > > Key: HBASE-8226 > URL: https://issues.apache.org/jira/browse/HBASE-8226 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 8226-branch-0.94.patch, 8226.patch, 8226.patch > > > HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region > it is counting belongs to the table created by the test and will not return > if it accidentally counts "too many" regions, for example the regions of the > ACL table when security is enabled. > Made an attempt at fixing this as part of HBASE-8209 but broke > TestMasterTransitions. Try again here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions
[ https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8226: -- Attachment: 8226.patch Attach trunk patch again and resubmit. > HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts > "too many" regions > > > Key: HBASE-8226 > URL: https://issues.apache.org/jira/browse/HBASE-8226 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 8226-branch-0.94.patch, 8226.patch, 8226.patch > > > HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region > it is counting belongs to the table created by the test and will not return > if it accidentally counts "too many" regions, for example the regions of the > ACL table when security is enabled. > Made an attempt at fixing this as part of HBASE-8209 but broke > TestMasterTransitions. Try again here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions
[ https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8226: -- Status: Patch Available (was: Open) > HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts > "too many" regions > > > Key: HBASE-8226 > URL: https://issues.apache.org/jira/browse/HBASE-8226 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Attachments: 8226-branch-0.94.patch, 8226.patch, 8226.patch > > > HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region > it is counting belongs to the table created by the test and will not return > if it accidentally counts "too many" regions, for example the regions of the > ACL table when security is enabled. > Made an attempt at fixing this as part of HBASE-8209 but broke > TestMasterTransitions. Try again here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-7970:
Attachment: HBASE-7970-v0.patch
Here's the patch. I ended up encapsulated bunch of things, so r is at
https://reviews.apache.org/r/10202/
> Improve file descriptor usage: currently, there are two file descriptors per
> storefile
> --
>
> Key: HBASE-7970
> URL: https://issues.apache.org/jira/browse/HBASE-7970
> Project: HBase
> Issue Type: Sub-task
>Reporter: Himanshu Vashishtha
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7970-v0.patch
>
>
> This is because there are two open calls in the HFile: one with checksum and
> another for without checksum support in v2:
> see the method in HFile:createReaderWithEncoding()
> {code}
> FSDataInputStream fsdis = fs.open(path);
> FSDataInputStream fsdisNoFsChecksum = fsdis;
> // If the fs is not an instance of HFileSystem, then create an
> // instance of HFileSystem that wraps over the specified fs.
> // In this case, we will not be able to avoid checksumming inside
> // the filesystem.
> if (!(fs instanceof HFileSystem)) {
> hfs = new HFileSystem(fs);
> } else {
> hfs = (HFileSystem)fs;
> // open a stream to read data without checksum verification in
> // the filesystem
> fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-7970:
Status: Patch Available (was: Open)
> Improve file descriptor usage: currently, there are two file descriptors per
> storefile
> --
>
> Key: HBASE-7970
> URL: https://issues.apache.org/jira/browse/HBASE-7970
> Project: HBase
> Issue Type: Sub-task
>Reporter: Himanshu Vashishtha
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7970-v0.patch
>
>
> This is because there are two open calls in the HFile: one with checksum and
> another for without checksum support in v2:
> see the method in HFile:createReaderWithEncoding()
> {code}
> FSDataInputStream fsdis = fs.open(path);
> FSDataInputStream fsdisNoFsChecksum = fsdis;
> // If the fs is not an instance of HFileSystem, then create an
> // instance of HFileSystem that wraps over the specified fs.
> // In this case, we will not be able to avoid checksumming inside
> // the filesystem.
> if (!(fs instanceof HFileSystem)) {
> hfs = new HFileSystem(fs);
> } else {
> hfs = (HFileSystem)fs;
> // open a stream to read data without checksum verification in
> // the filesystem
> fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts "too many" regions
[
https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617816#comment-13617816
]
Hadoop QA commented on HBASE-8226:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576186/8226-branch-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5056//console
This message is automatically generated.
> HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts
> "too many" regions
>
>
> Key: HBASE-8226
> URL: https://issues.apache.org/jira/browse/HBASE-8226
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 0.95.0, 0.96.0, 0.94.7
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Attachments: 8226-branch-0.94.patch, 8226.patch
>
>
> HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region
> it is counting belongs to the table created by the test and will not return
> if it accidentally counts "too many" regions, for example the regions of the
> ACL table when security is enabled.
> Made an attempt at fixing this as part of HBASE-8209 but broke
> TestMasterTransitions. Try again here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7977) Online merge should acquire table lock
[ https://issues.apache.org/jira/browse/HBASE-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617809#comment-13617809 ] Hudson commented on HBASE-7977: --- Integrated in hbase-0.95-on-hadoop2 #47 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/47/]) HBASE-7977 Online merge should acquire table lock (Revision 1462386) Result = FAILURE enis : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeRequest.java > Online merge should acquire table lock > -- > > Key: HBASE-7977 > URL: https://issues.apache.org/jira/browse/HBASE-7977 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment, regionserver >Affects Versions: 0.95.0, 0.98.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.95.0, 0.98.0 > > Attachments: hbase-7977_v1.patch > > > Once online merge (HBASE-7403) is in, we should ensure that we acquire a > table write lock during the merge. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8207) Replication could have data loss when machine name contains hyphen "-"
[
https://issues.apache.org/jira/browse/HBASE-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617807#comment-13617807
]
Hudson commented on HBASE-8207:
---
Integrated in hbase-0.95-on-hadoop2 #47 (See
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/47/])
HBASE-8207 Addendum fixes the typo in extractDeadServersFromZNodeString()
(Revision 1462553)
HBASE-8207 Replication could have data loss when machine name contains hyphen
"-" (Jeffrey) (Revision 1462516)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
tedyu :
Files :
*
/hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
> Replication could have data loss when machine name contains hyphen "-"
> --
>
> Key: HBASE-8207
> URL: https://issues.apache.org/jira/browse/HBASE-8207
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 0.95.0, 0.94.6
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.95.0, 0.98.0, 0.94.7
>
> Attachments: 8207-trunk-addendum.txt, 8207_v3.patch, failed.txt,
> hbase-8207-0.94-v1.patch, hbase-8207.patch, hbase-8207_v1.patch,
> hbase-8207_v2.patch, hbase-8207_v2.patch, HBASE-8212-94.patch
>
>
> In the recent test case TestReplication* failures, I'm finally able to find
> the cause(or one of causes) for its intermittent failures.
> When a machine name contains "-", it breaks the function
> ReplicationSource.checkIfQueueRecovered. It causes the following issue:
> deadRegionServers list is way off so that replication doesn't wait for log
> splitting finish for a wal file and move on to the next one(data loss)
> You can see that replication use those weird paths constructed from
> deadRegionServers to check a file existence
> {code}
> 2013-03-26 21:26:51,385 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,386 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/1.compute.internal,52170,1364333181125-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,387 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,389 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/west-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,391 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,394 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/156.us-splitting/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,396 INFO
> [ReplicationExecutor-0.replicationSource,2-ip-10-197-0-156.us-west-1.compute.internal,52170,1364333181125]
> regionserver.ReplicationSource(524): Possible location
> hdfs://localhost:52882/user/ec2-user/hbase/.logs/0/ip-10-197-0-156.us-west-1.compute.internal%252C52170%252C1364333181125.1364333199540
> 2013-03-26 21:26:51,398 INFO
> [ReplicationExecutor-0.replicationSource,2-ip
[jira] [Commented] (HBASE-8187) trunk/0.95 tarball packaging
[ https://issues.apache.org/jira/browse/HBASE-8187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617808#comment-13617808 ] Hudson commented on HBASE-8187: --- Integrated in hbase-0.95-on-hadoop2 #47 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/47/]) HBASE-8187 trunk/0.95 tarball packaging; ADDENDUM for 0.95 (Revision 1462620) HBASE-8187 trunk/0.95 tarball packaging (Revision 1462608) Result = FAILURE stack : Files : * /hbase/branches/0.95/hbase-assembly/pom.xml * /hbase/branches/0.95/hbase-client/pom.xml * /hbase/branches/0.95/hbase-common/pom.xml * /hbase/branches/0.95/hbase-examples/pom.xml * /hbase/branches/0.95/hbase-hadoop-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop1-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop2-compat/pom.xml * /hbase/branches/0.95/hbase-it/pom.xml * /hbase/branches/0.95/hbase-prefix-tree/pom.xml * /hbase/branches/0.95/hbase-protocol/pom.xml * /hbase/branches/0.95/hbase-server/pom.xml * /hbase/branches/0.95/pom.xml stack : Files : * /hbase/branches/0.95/hbase-assembly * /hbase/branches/0.95/hbase-assembly/pom.xml * /hbase/branches/0.95/hbase-assembly/src * /hbase/branches/0.95/hbase-assembly/src/assembly * /hbase/branches/0.95/hbase-assembly/src/assembly/components.xml * /hbase/branches/0.95/hbase-assembly/src/assembly/hadoop-one-compat.xml * /hbase/branches/0.95/hbase-assembly/src/assembly/hadoop-two-compat.xml * /hbase/branches/0.95/hbase-assembly/src/assembly/src.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx * /hbase/branches/0.95/hbase-assembly/src/docbkx/book.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/case_studies.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/community.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/configuration.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/customization.xsl * /hbase/branches/0.95/hbase-assembly/src/docbkx/developer.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/external_apis.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/getting_started.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/ops_mgt.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/performance.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/preface.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/rpc.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/security.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/shell.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/troubleshooting.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/upgrading.xml * /hbase/branches/0.95/hbase-assembly/src/docbkx/zookeeper.xml * /hbase/branches/0.95/hbase-assembly/src/site * /hbase/branches/0.95/hbase-assembly/src/site/resources * /hbase/branches/0.95/hbase-assembly/src/site/resources/css * /hbase/branches/0.95/hbase-assembly/src/site/resources/css/freebsd_docbook.css * /hbase/branches/0.95/hbase-assembly/src/site/resources/css/site.css * /hbase/branches/0.95/hbase-assembly/src/site/resources/doap_Hbase.rdf * /hbase/branches/0.95/hbase-assembly/src/site/resources/images * /hbase/branches/0.95/hbase-assembly/src/site/resources/images/big_h_logo.svg * /hbase/branches/0.95/hbase-assembly/src/site/resources/images/hbase_logo.svg * /hbase/branches/0.95/hbase-assembly/src/site/site.vm * /hbase/branches/0.95/hbase-assembly/src/site/site.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/acid-semantics.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/bulk-loads.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/cygwin.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/index.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/metrics.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/old_news.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/pseudo-distributed.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/replication.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/resources.xml * /hbase/branches/0.95/hbase-assembly/src/site/xdoc/sponsors.xml * /hbase/branches/0.95/hbase-assembly/src/xslt * /hbase/branches/0.95/hbase-assembly/src/xslt/configuration_to_docbook_section.xsl * /hbase/branches/0.95/hbase-client/pom.xml * /hbase/branches/0.95/hbase-common/pom.xml * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-examples/pom.xml * /hbase/branches/0.95/hbase-hadoop-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop1-compat/pom.xml * /hbase/branches/0.95/hbase-hadoop2-compat/pom.xml * /hbase/branches/0.95/hbase-it/pom.xml * /hbase/branches/0.95/hbase-prefix-tree/pom.xml * /hbase/branches/0.95/hbase-protocol/pom.xml * /hbase/branches/0.95/hbase-server/pom.xml * /hbase/branches/0.95/pom.xml * /hbase/branches/0.95/src > trunk/0.95 tarball packaging > > > Key: HBASE-8187 > URL: https://issues.apache
[jira] [Commented] (HBASE-8209) Improve LoadTest extensibility
[ https://issues.apache.org/jira/browse/HBASE-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13617806#comment-13617806 ] Hudson commented on HBASE-8209: --- Integrated in hbase-0.95-on-hadoop2 #47 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/47/]) Amend HBASE-8209. Revert changes to HBaseTestingUtility#waitUntilAllRegionsAssigned (Revision 1462640) HBASE-8209. Improve LoadTest extensibility (Revision 1462630) Result = FAILURE apurtell : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java apurtell : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java > Improve LoadTest extensibility > -- > > Key: HBASE-8209 > URL: https://issues.apache.org/jira/browse/HBASE-8209 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.95.0, 0.96.0, 0.94.7 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.95.0, 0.96.0, 0.94.7 > > Attachments: addendum-0.94.patch, addendum-trunk.patch, > HBASE-8209-0.94.patch, HBASE-8209-0.94-v2.patch, HBASE-8209-0.94-v3.patch, > HBASE-8209-trunk.patch, HBASE-8209-trunk-v2.patch, HBASE-8209-trunk-v3.patch > > > This patch rolls up related changes: > - Allow classes extending TestMiniClusterLoadSequential to override how > reader and writer threads are set up, including extending the reader and > writers themselves. > - Make it so classes extending MultiThreadedWriter can cleanly override how > Puts are constructed. > - Provide an option for passing in a custom table descriptor to > HBaseTestingUtility#createPreSplitLoadTestTable. > > - HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the > region it is counting belongs to the table created by the test and will not > return if it accidentally counts "too many" regions, for example the regions > of the ACL table when security is enabled. (Some class based on > TestMiniClusterLoadSequential may want to enable security.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
