[jira] [Commented] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458463#comment-13458463
]
Hudson commented on HBASE-6834:
---
Integrated in HBase-TRUNK #3352 (See
[https://builds.apache.org/job/HBase-TRUNK/3352/])
HBASE-6834 HBaseClusterManager shell code breaks hadoop-2.0 build (Revision
1387458)
Result = FAILURE
stack :
Files :
*
/hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/HBaseClusterManager.java
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
> Fix For: 0.96.0
>
> Attachments: hbase-6834_v1.patch
>
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458462#comment-13458462 ] Hadoop QA commented on HBASE-5498: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12545667/HBASE-5498_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 16 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2896//console This message is automatically generated. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: mapred, security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0, 0.94.3 > > Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, > HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458443#comment-13458443 ] Jesse Yates commented on HBASE-6610: [~saint@gmail.com] commented on RB - almost there, mostly nits (and possibly some matter of the late hour of the review). do we want to roll this into the 0.96 or the snapshots branch? Seems like its only being used in snapshots, but given the breadth of code it touches, I'd be ok with it going into trunk and then ported over to snapshots (or vice-versa, as long as it happens in a short period of time). > HFileLink: Hardlink alternative for snapshot restore > > > Key: HBASE-6610 > URL: https://issues.apache.org/jira/browse/HBASE-6610 > Project: HBase > Issue Type: Sub-task > Components: io >Affects Versions: 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Labels: snapshot > Fix For: 0.96.0 > > Attachments: HBASE-6610-v1.patch, HBASE-6610-v2.patch, > HBASE-6610-v3.patch, HBASE-6610-v5.patch, HBASE-6610-v6.patch > > > To avoid copying data during restore snapshot we need to introduce an HFile > Link that allows to reference a file that can be in the original path > (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive > directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
[ https://issues.apache.org/jira/browse/HBASE-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458439#comment-13458439 ] Elliott Clark commented on HBASE-6833: -- Would taking a currentTimeMillis and the nanoTime at class instantiation allow us to create an always increasing wall clock time. we would have a startNano and a startMilli return startMili + toMilli(startNano - System.nanoTime()); That has the potential to drift a little bit from wall time but should still be pretty close. And if we really need to stay close to wall time we can have a repeating timer that resets startMilli and startNano. > [WINDOWS] Java Milisecond precision on Windows > -- > > Key: HBASE-6833 > URL: https://issues.apache.org/jira/browse/HBASE-6833 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > HBase relies on the system clock obtained by System.currentTimeMilis() to > supply the version, if it is not provided by the client in Put's and > Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, > the milis clock might not be updated every milisecond, which might cause > unexpected behavior. We also did some preliminary analysis there. > In this issue, we can further discuss and inspect whether it is a problem for > main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458432#comment-13458432
]
Lars Hofhansl commented on HBASE-6698:
--
Super minor nit:
{code}
+Pair mutateWithLocks[] = new Pair[1];
+mutateWithLocks[0] = new Pair(mutation, lid);
{code}
can be written as:
{code}
Pair mutateWithLocks[] = new Pair[] {new
Pair(mutation, lid)};
{code}
Also, this method should be marked with:
{{@SuppressWarnings("unchecked")}}
> Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
> --
>
> Key: HBASE-6698
> URL: https://issues.apache.org/jira/browse/HBASE-6698
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
> Fix For: 0.96.0
>
> Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch,
> HBASE-6698_3.patch, HBASE-6698_5.patch, HBASE-6698_6.patch,
> HBASE-6698_6.patch, HBASE-6698_6.patch, HBASE-6698_6.patch, HBASE-6698.patch
>
>
> Currently the checkAndPut and checkAndDelete api internally calls the
> internalPut and internalDelete. May be we can just call doMiniBatchMutation
> only. This will help in future like if we have some hooks and the CP
> handles certain cases in the doMiniBatchMutation the same can be done while
> doing a put thro checkAndPut or while doing a delete thro checkAndDelete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-6834.
--
Resolution: Fixed
Fix Version/s: 0.96.0
Hadoop Flags: Reviewed
Committed to trunk. Thanks Enis and G.
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
> Fix For: 0.96.0
>
> Attachments: hbase-6834_v1.patch
>
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6649: --- Attachment: (was: 6649-fix-io-exception-handling.patch) > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-fix-io-exception-handling.patch, 6649-trunk.patch, 6649-trunk.patch, > 6649.txt, HBase-0.92 #495 test - queueFailover [Jenkins].html, HBase-0.92 > #502 test - queueFailover [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6649: --- Attachment: 6649-fix-io-exception-handling.patch Attaching a more complete fix (for 0.94) [~jdcryans], could you please try this patch out in your cluster. The more I think about it, the more I am beginning to believe that setting the position so that it always points to a valid location is the fix here... [~lhofhansl] I have seen dataloss issues (via the unit test) without this patch.. > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-fix-io-exception-handling.patch, 6649-fix-io-exception-handling.patch, > 6649-trunk.patch, 6649-trunk.patch, 6649.txt, HBase-0.92 #495 test - > queueFailover [Jenkins].html, HBase-0.92 #502 test - queueFailover > [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6837) RegionServer Groups corpcoessor apis
[ https://issues.apache.org/jira/browse/HBASE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu reassigned HBASE-6837: -- Assignee: Francis Liu > RegionServer Groups corpcoessor apis > > > Key: HBASE-6837 > URL: https://issues.apache.org/jira/browse/HBASE-6837 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Francis Liu > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6837) RegionServer Groups corpcoessor apis
Francis Liu created HBASE-6837: -- Summary: RegionServer Groups corpcoessor apis Key: HBASE-6837 URL: https://issues.apache.org/jira/browse/HBASE-6837 Project: HBase Issue Type: Sub-task Reporter: Francis Liu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458423#comment-13458423
]
stack commented on HBASE-6698:
--
Sorry lads. Distracted.
+1 on committing this patch. It removes two fat delete and put methods and
instead has us go by the batch mutate path. Nice improvement.
Below are some comments. You can address on commit Ram.
Why we do this?
{code}
-Delete delete = new Delete();
+Delete delete = new Delete(new byte[0]);
{code}
Null row in Delete is handled differently to a row that is an empty byte array
(Would suggest HConstants.EMPTY_BYTE_ARRAY instead of creating a new byte array
each time through).
Is this cast necessary?
{code}
+ doBatchMutate((Mutation)w, lid);
{code}
This comment should be removed? Its going to seem funny when its not followed
by Put and Delete stuff:
{code}
// Using default cluster id, as this can only happen in the
// originating cluster. A slave cluster receives the result as a Put
// or Delete
{code}
Here you put the square brackets after the variable name:
{code}
+Pair mutateWithLocks[] = new Pair[1];
{code}
In the rest of the method, the square brackets precede the variable name.
Would suggest you be consistent.
> Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
> --
>
> Key: HBASE-6698
> URL: https://issues.apache.org/jira/browse/HBASE-6698
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
> Fix For: 0.96.0
>
> Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch,
> HBASE-6698_3.patch, HBASE-6698_5.patch, HBASE-6698_6.patch,
> HBASE-6698_6.patch, HBASE-6698_6.patch, HBASE-6698_6.patch, HBASE-6698.patch
>
>
> Currently the checkAndPut and checkAndDelete api internally calls the
> internalPut and internalDelete. May be we can just call doMiniBatchMutation
> only. This will help in future like if we have some hooks and the CP
> handles certain cases in the doMiniBatchMutation the same can be done while
> doing a put thro checkAndPut or while doing a delete thro checkAndDelete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6723) Implement RegionServer Group Based Balancer
[ https://issues.apache.org/jira/browse/HBASE-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-6723: --- Description: Re-purposing this Jira after the discussion last week. Assignee: Vandana Ayyalasomayajula Summary: Implement RegionServer Group Based Balancer (was: Make AssignmentManager pluggable) > Implement RegionServer Group Based Balancer > --- > > Key: HBASE-6723 > URL: https://issues.apache.org/jira/browse/HBASE-6723 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Vandana Ayyalasomayajula > > Re-purposing this Jira after the discussion last week. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
[ https://issues.apache.org/jira/browse/HBASE-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458419#comment-13458419 ] Lars Hofhansl commented on HBASE-6833: -- We need to keep in mind then that nanotime does not reflect any absolute time and is only supposed to be used for time-differences. As long we make sure it'll only ever used for that, we should be OK. I.e. we could not take nanotime, and use that for the persistent KV timestamps. (Going the documentation and what I have read elsewhere) > [WINDOWS] Java Milisecond precision on Windows > -- > > Key: HBASE-6833 > URL: https://issues.apache.org/jira/browse/HBASE-6833 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > HBase relies on the system clock obtained by System.currentTimeMilis() to > supply the version, if it is not provided by the client in Put's and > Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, > the milis clock might not be updated every milisecond, which might cause > unexpected behavior. We also did some preliminary analysis there. > In this issue, we can further discuss and inspect whether it is a problem for > main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Fix Version/s: 0.94.3 Status: Patch Available (was: Reopened) > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: mapred, security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0, 0.94.3 > > Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, > HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_94.patch > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: mapred, security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0 > > Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, > HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_trunk.patch patches including unit tests and some changes. Notice that the trunk patch is a rebase of 0.94 for simplicity. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: mapred, security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.96.0 > > Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, > HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458404#comment-13458404 ] Lars Hofhansl commented on HBASE-6649: -- I say we revert from 0.94.2 and retry in 0.94.3. Although from DD's comment: bq. If the second call (within the while loop) throws an exception (like EOFException), it basically destroys the work done up until then. Therefore, some rows would never be replicated. This would be a dataloss issue without the fix. I find that a bit confusion. Since J-D saw the ignored exception in the test cluster eventually on all machines, it seems there was data lost in all versions before 0.94.2? That seems very unlikely. > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-fix-io-exception-handling.patch, 6649-trunk.patch, 6649-trunk.patch, > 6649.txt, HBase-0.92 #495 test - queueFailover [Jenkins].html, HBase-0.92 > #502 test - queueFailover [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458402#comment-13458402
]
Gregory Chanan commented on HBASE-6834:
---
+1
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
> Attachments: hbase-6834_v1.patch
>
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6408) Naming and documenting of the hadoop-metrics2.properties file
[ https://issues.apache.org/jira/browse/HBASE-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458401#comment-13458401 ] Hudson commented on HBASE-6408: --- Integrated in HBase-TRUNK #3351 (See [https://builds.apache.org/job/HBase-TRUNK/3351/]) HBASE-6408 Naming and documenting of the hadoop-metrics2.properties file (Revision 1387443) Result = FAILURE stack : Files : * /hbase/trunk/conf/hadoop-metrics2-hbase.properties * /hbase/trunk/conf/hadoop-metrics2.properties > Naming and documenting of the hadoop-metrics2.properties file > - > > Key: HBASE-6408 > URL: https://issues.apache.org/jira/browse/HBASE-6408 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6408-0.patch > > > hadoop-metrics2.properties is currently where metrics2 loads it's sinks. > This file could be better named, hadoop-hbase-metrics2.properties > In addition it needs examples like the current hadoop-metrics.properties has. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
[ https://issues.apache.org/jira/browse/HBASE-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458392#comment-13458392 ] Elliott Clark commented on HBASE-6833: -- Yeah. nanoTime seems like it would work. Xp and earlier are the ones I remember having 15ms times; I don't remember the server versions as well since I never had to deal with them. So I'm kind of interested in what environment Enis is seeing 10ms+ > [WINDOWS] Java Milisecond precision on Windows > -- > > Key: HBASE-6833 > URL: https://issues.apache.org/jira/browse/HBASE-6833 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > HBase relies on the system clock obtained by System.currentTimeMilis() to > supply the version, if it is not provided by the client in Put's and > Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, > the milis clock might not be updated every milisecond, which might cause > unexpected behavior. We also did some preliminary analysis there. > In this issue, we can further discuss and inspect whether it is a problem for > main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6813) Optimise the time spent holding the updateLock under log roll
[ https://issues.apache.org/jira/browse/HBASE-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458390#comment-13458390 ] Jean-Daniel Cryans commented on HBASE-6813: --- We actually create the file outside of that lock, so it's cleanupCurrentWriter that needs to be broken up. > Optimise the time spent holding the updateLock under log roll > - > > Key: HBASE-6813 > URL: https://issues.apache.org/jira/browse/HBASE-6813 > Project: HBase > Issue Type: Improvement >Reporter: Amitanand Aiyer >Priority: Minor > > Log roll entails syncing the old log, closing it and creating a new log file. > We currently do all the 3 steps while holding the updateLock. This causes > latency spikes for puts during this time. > We only need to sync the old log under the lock. Creating the new file, can > be done before grabbing the lock. Closing the old file can be done after we > release the lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6408) Naming and documenting of the hadoop-metrics2.properties file
[ https://issues.apache.org/jira/browse/HBASE-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6408: - Fix Version/s: 0.96.0 > Naming and documenting of the hadoop-metrics2.properties file > - > > Key: HBASE-6408 > URL: https://issues.apache.org/jira/browse/HBASE-6408 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6408-0.patch > > > hadoop-metrics2.properties is currently where metrics2 loads it's sinks. > This file could be better named, hadoop-hbase-metrics2.properties > In addition it needs examples like the current hadoop-metrics.properties has. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6408) Naming and documenting of the hadoop-metrics2.properties file
[ https://issues.apache.org/jira/browse/HBASE-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6408: - Resolution: Fixed Release Note: The metrics config file was conf/hadoop-metrics2-hbase.properties and is now conf/hadoop-metrics2.properties Status: Resolved (was: Patch Available) Applied to trunk. Thanks Elliott. > Naming and documenting of the hadoop-metrics2.properties file > - > > Key: HBASE-6408 > URL: https://issues.apache.org/jira/browse/HBASE-6408 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Attachments: HBASE-6408-0.patch > > > hadoop-metrics2.properties is currently where metrics2 loads it's sinks. > This file could be better named, hadoop-hbase-metrics2.properties > In addition it needs examples like the current hadoop-metrics.properties has. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6591) checkAndPut executed/not metrics
[ https://issues.apache.org/jira/browse/HBASE-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458388#comment-13458388 ] stack commented on HBASE-6591: -- +1 Looks like there are a couple of tests failing in hadoop 2.0 w/ a while. > checkAndPut executed/not metrics > > > Key: HBASE-6591 > URL: https://issues.apache.org/jira/browse/HBASE-6591 > Project: HBase > Issue Type: Task > Components: metrics, regionserver >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-6591.patch, HBASE-6591-v2.patch, > HBASE-6591-v2.patch, HBASE-6591-v3.patch > > > checkAndPut/checkAndDelete return true if the new put was executed, false > otherwise. > So clients can figure out this metric for themselves, but it would be useful > to get a look at what is happening on the cluster as a whole, across all > clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
[ https://issues.apache.org/jira/browse/HBASE-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458385#comment-13458385 ] stack commented on HBASE-6833: -- So a WindowsEnvironmentEdge backed by nano time? > [WINDOWS] Java Milisecond precision on Windows > -- > > Key: HBASE-6833 > URL: https://issues.apache.org/jira/browse/HBASE-6833 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > HBase relies on the system clock obtained by System.currentTimeMilis() to > supply the version, if it is not provided by the client in Put's and > Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, > the milis clock might not be updated every milisecond, which might cause > unexpected behavior. We also did some preliminary analysis there. > In this issue, we can further discuss and inspect whether it is a problem for > main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6814) [WINDOWS] HBase on Windows
[ https://issues.apache.org/jira/browse/HBASE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458383#comment-13458383 ] stack commented on HBASE-6814: -- Up to you. Going by what you've filed so far, patches seem small enough and pointed (except the millisecond issue). Maybe 3. would be the way to go? It'd be silly having a 'windows' component rather than WINDOWS prefix on issues? > [WINDOWS] HBase on Windows > -- > > Key: HBASE-6814 > URL: https://issues.apache.org/jira/browse/HBASE-6814 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > This is an umbrella jira to support windows as a first class citizen. > The short term goals are: > - Get unit tests passing > - Scripts converted to windows > - Runtime does not depend on cgywin (build can still depend on cygwin for > now) > - Hbase on windows will consume hadoop branch-1-win artifacts. > - Get nightly jenkins build on windows -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
[ https://issues.apache.org/jira/browse/HBASE-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458382#comment-13458382 ] Elliott Clark commented on HBASE-6833: -- Windows can have a more accurate normal system clock however it causes a lot of cpu load so is not always done: See: http://msdn.microsoft.com/en-us/library/dd757624%28VS.85%29.aspx Calling that works but it's frowned upon unless you really need timers that are < 10 ms resolution. https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks Seems to suggest that using the System.nanoTime will give the correct time as determined by QueryPerformanceFrequency. So our timing can use nanoTime converted to ms. This will only work on modernish hardware and modernish windows. However I think that's probably good enough. That still leaves the timers. Are there any cases where our timers must be 1ms accurate ? If it is then the above link seems to suggest that setting a timer that's not a multiple of 10ms will cause java to set the clock timer to 1 ms (though on older hardware I seem to remember that the api is ignored). > [WINDOWS] Java Milisecond precision on Windows > -- > > Key: HBASE-6833 > URL: https://issues.apache.org/jira/browse/HBASE-6833 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > HBase relies on the system clock obtained by System.currentTimeMilis() to > supply the version, if it is not provided by the client in Put's and > Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, > the milis clock might not be updated every milisecond, which might cause > unexpected behavior. We also did some preliminary analysis there. > In this issue, we can further discuss and inspect whether it is a problem for > main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6835) HBaseAdmin.flush claims to be asynchronous but appears to be synchronous
[
https://issues.apache.org/jira/browse/HBASE-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458381#comment-13458381
]
Jean-Daniel Cryans commented on HBASE-6835:
---
Related or similar to HBASE-4198?
> HBaseAdmin.flush claims to be asynchronous but appears to be synchronous
>
>
> Key: HBASE-6835
> URL: https://issues.apache.org/jira/browse/HBASE-6835
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
>
> Relevant comment:
> {code}
>* Flush a table or an individual region.
>* Asynchronous operation.
> {code}
> but it looks like it's synchronous. In fact, it returns whether the flush
> ran or not:
> {code}
> message FlushRegionResponse {
> required uint64 lastFlushTime = 1;
> optional bool flushed = 2;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458378#comment-13458378 ] Jean-Daniel Cryans commented on HBASE-6649: --- bq. The patch only catches and ignores IOE (as opposed to all exceptions) What it does do is permitting to read up to the end of the file. bq. [Not sure which hadoop version you are on, but there is no chance you are hitting HDFS-1108, right?] We are on CDH3u3, didn't change when we applied the patch. bq. Okay a plausible explanation - It's plausible but unless we really understand what that "gibberish" is at the end of the file, we can't truly make a fix. I don't know why that IOE is throw out but normally we just silently finish reading from the file. There is some special case here. bq. Should we pull HBASE-6719 into this? I think it's separate issues. > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-fix-io-exception-handling.patch, 6649-trunk.patch, 6649-trunk.patch, > 6649.txt, HBase-0.92 #495 test - queueFailover [Jenkins].html, HBase-0.92 > #502 test - queueFailover [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6827) [WINDOWS] TestScannerTimeout fails expecting a timeout
[
https://issues.apache.org/jira/browse/HBASE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458380#comment-13458380
]
stack commented on HBASE-6827:
--
Want to setup a windows build on jenkins? Is there one up on apache jenkins?
There must be?
> [WINDOWS] TestScannerTimeout fails expecting a timeout
> --
>
> Key: HBASE-6827
> URL: https://issues.apache.org/jira/browse/HBASE-6827
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> TestScannerTimeout.test2481() fails with:
> {code}
> java.lang.AssertionError: We should be timing out
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.hadoop.hbase.client.TestScannerTimeout.test2481(TestScannerTimeout.java:117)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6827) [WINDOWS] TestScannerTimeout fails expecting a timeout
[
https://issues.apache.org/jira/browse/HBASE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458379#comment-13458379
]
stack commented on HBASE-6827:
--
Excellent. Fixing more of our flakey tests.
> [WINDOWS] TestScannerTimeout fails expecting a timeout
> --
>
> Key: HBASE-6827
> URL: https://issues.apache.org/jira/browse/HBASE-6827
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> TestScannerTimeout.test2481() fails with:
> {code}
> java.lang.AssertionError: We should be timing out
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.hadoop.hbase.client.TestScannerTimeout.test2481(TestScannerTimeout.java:117)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6831) [WINDOWS] HBaseTestingUtility.expireSession() does not expire zookeeper session
[ https://issues.apache.org/jira/browse/HBASE-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458377#comment-13458377 ] stack commented on HBASE-6831: -- This is a good one. You are uncovering why some tests are sometimes flakey. So, add in a noop that goes against the server after the construction and before close? > [WINDOWS] HBaseTestingUtility.expireSession() does not expire zookeeper > session > --- > > Key: HBASE-6831 > URL: https://issues.apache.org/jira/browse/HBASE-6831 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > TestReplicationPeer fails because it forces the zookeeper session expiration > by calling HBaseTestingUtilty.expireSesssion(), but that function fails to do > so. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
[ https://issues.apache.org/jira/browse/HBASE-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458371#comment-13458371 ] stack commented on HBASE-6833: -- This one is a bit of a bummer. As you suggest elsewhere, in tests, could user EnvironmentEdge thing that updates each time its called but what happens when you run in production? You are going to get odd results. This does bring to the fore how much we presume a forward moving clock (Those spanner time servers with their aerials for GPS and atomic clock cards start to make a bit more sense now) > [WINDOWS] Java Milisecond precision on Windows > -- > > Key: HBASE-6833 > URL: https://issues.apache.org/jira/browse/HBASE-6833 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > HBase relies on the system clock obtained by System.currentTimeMilis() to > supply the version, if it is not provided by the client in Put's and > Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, > the milis clock might not be updated every milisecond, which might cause > unexpected behavior. We also did some preliminary analysis there. > In this issue, we can further discuss and inspect whether it is a problem for > main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458367#comment-13458367
]
stack commented on HBASE-6834:
--
+1 on the patch. What you think Gregory (Given its a one liner maybe we put
off pulling in the Shell* classes?)
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
> Attachments: hbase-6834_v1.patch
>
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-6834:
-
Attachment: hbase-6834_v1.patch
Attaching a one liner.
{code}
MAVEN_OPTS=-Xmx1024m mvn clean install -DskipTests -Dhadoop.profile=2.0
-Dhadoop.version=2.0.1-alpha
{code}
seems to work.
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
> Attachments: hbase-6834_v1.patch
>
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6836) [89-fb] Parallel deletes in HBase Thrift server
Mikhail Bautin created HBASE-6836: - Summary: [89-fb] Parallel deletes in HBase Thrift server Key: HBASE-6836 URL: https://issues.apache.org/jira/browse/HBASE-6836 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin We need to expose server-side parallel batch deletes through the Thrift server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458357#comment-13458357
]
Enis Soztutar commented on HBASE-6834:
--
Shell and ShellCommandExecutor are non-trivial classes. I am not sure we should
just copy them. If we run into this kind of problems in the future, we might
develop our own, or maybe use the shims layer. wdyt?
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458352#comment-13458352 ] Lars Hofhansl commented on HBASE-6649: -- Should we pull HBASE-6719 into this? > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-fix-io-exception-handling.patch, 6649-trunk.patch, 6649-trunk.patch, > 6649.txt, HBase-0.92 #495 test - queueFailover [Jenkins].html, HBase-0.92 > #502 test - queueFailover [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6699) Setting username in Connection in non-secure HBase
[ https://issues.apache.org/jira/browse/HBASE-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6699: - Fix Version/s: (was: 0.94.3) (was: 0.92.3) > Setting username in Connection in non-secure HBase > -- > > Key: HBASE-6699 > URL: https://issues.apache.org/jira/browse/HBASE-6699 > Project: HBase > Issue Type: Improvement > Components: ipc >Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.94.1 >Reporter: Himanshu Vashishtha >Assignee: Himanshu Vashishtha > Attachments: HBase-6699-v1.patch > > > We recently had a requirement where we need to log the information about > various users who were using non-secure HBase cluster. > The user level logging is supported as part of security, but in 0.92, 0.94 > security related code is separate. This jira is about adding that support in > non-secure code. > This feature is already there in trunk, after we merge the security related > code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458351#comment-13458351
]
Gregory Chanan commented on HBASE-6834:
---
you think we should just change the signature, not bring in the class?
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6720) Optionally limit number of regions balanced in each balancer run
[ https://issues.apache.org/jira/browse/HBASE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6720: - Fix Version/s: (was: 0.94.3) (was: 0.96.0) > Optionally limit number of regions balanced in each balancer run > > > Key: HBASE-6720 > URL: https://issues.apache.org/jira/browse/HBASE-6720 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 6720-0.96-v1.txt > > > See discussion on HBASE-3866 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6835) HBaseAdmin.flush claims to be asynchronous but appears to be synchronous
Gregory Chanan created HBASE-6835:
-
Summary: HBaseAdmin.flush claims to be asynchronous but appears to
be synchronous
Key: HBASE-6835
URL: https://issues.apache.org/jira/browse/HBASE-6835
Project: HBase
Issue Type: Bug
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
Relevant comment:
{code}
* Flush a table or an individual region.
* Asynchronous operation.
{code}
but it looks like it's synchronous. In fact, it returns whether the flush ran
or not:
{code}
message FlushRegionResponse {
required uint64 lastFlushTime = 1;
optional bool flushed = 2;
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6695) [Replication] Data will lose if RegionServer down during transferqueue
[ https://issues.apache.org/jira/browse/HBASE-6695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6695: - Fix Version/s: (was: 0.94.3) (was: 0.96.0) > [Replication] Data will lose if RegionServer down during transferqueue > -- > > Key: HBASE-6695 > URL: https://issues.apache.org/jira/browse/HBASE-6695 > Project: HBase > Issue Type: Bug > Components: replication >Affects Versions: 0.94.1 >Reporter: terry zhang >Priority: Critical > Attachments: HBASE-6695-4trunk.patch, HBASE-6695-4trunk_v2.patch, > HBASE-6695-4trunk_v3.patch, HBASE-6695.patch > > > When we ware testing Replication failover feature we found if we kill a > regionserver during it transferqueue ,we found only part of the hlog znode > copy to the right path because failover process is interrupted. > Log: > 2012-08-29 12:20:05,660 INFO > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager: > Moving dw92.kgb.sqa.cm4,60020,1346210789716's hlogs to my queue > 2012-08-29 12:20:05,765 DEBUG > org.apache.hadoop.hbase.replication.ReplicationZookeeper: Creating > dw92.kgb.sqa.cm4%2C60020%2C13462107 89716.1346213720708 with data 210508162 > 2012-08-29 12:20:05,850 DEBUG > org.apache.hadoop.hbase.replication.ReplicationZookeeper: Creating > dw92.kgb.sqa.cm4%2C60020%2C13462107 89716.1346213886800 with data > 2012-08-29 12:20:05,938 DEBUG > org.apache.hadoop.hbase.replication.ReplicationZookeeper: Creating > dw92.kgb.sqa.cm4%2C60020%2C1346210789716.1346213830559 with data > 2012-08-29 12:20:06,055 DEBUG > org.apache.hadoop.hbase.replication.ReplicationZookeeper: Creating > dw92.kgb.sqa.cm4%2C60020%2C1346210789716.1346213775146 with data > 2012-08-29 12:20:06,277 WARN > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Failed all from region=.ME > TA.,,1.1028785192, hostname=dw93.kgb.sqa.cm4, port=60020 > java.util.concurrent.ExecutionException: java.net.ConnectException: > Connection refused > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > .. > This server is down . > ZK node status: > [zk: 10.232.98.77:2181(CONNECTED) 6] ls > /hbase-test3-repl/replication/rs/dw92.kgb.sqa.cm4,60020,1346210789716 > [lock, 1, 1-dw89.kgb.sqa.cm4,60020,1346202436268] > > dw92 is down , but Node dw92.kgb.sqa.cm4,60020,1346210789716 can't be deleted -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458347#comment-13458347
]
Enis Soztutar commented on HBASE-6834:
--
Thanks for spotting this. I'll provide a patch for changing the signature.
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
[
https://issues.apache.org/jira/browse/HBASE-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar reassigned HBASE-6834:
Assignee: Enis Soztutar
> HBaseClusterManager shell code breaks hadoop-2.0 build
> --
>
> Key: HBASE-6834
> URL: https://issues.apache.org/jira/browse/HBASE-6834
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Enis Soztutar
>
> I get the following error:
> {noformat}
> HBaseClusterManager.java:[73,23] getExecString() in
> org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
> getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
> attempting to assign weaker access privileges; was public
> {noformat}
> the issue is that getExecString() is declared public in hadoop-2.0, but
> protected in hadoop-1.0. In HBase, it is declared protected, and you can't
> downgrade from public to protected.
> We can just declare the function public and that seems to work, but given
> that in hadoop the class is declared
> {code}
> @InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
> {code}
> perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6834) HBaseClusterManager shell code breaks hadoop-2.0 build
Gregory Chanan created HBASE-6834:
-
Summary: HBaseClusterManager shell code breaks hadoop-2.0 build
Key: HBASE-6834
URL: https://issues.apache.org/jira/browse/HBASE-6834
Project: HBase
Issue Type: Bug
Reporter: Gregory Chanan
I get the following error:
{noformat}
HBaseClusterManager.java:[73,23] getExecString() in
org.apache.hadoop.hbase.HBaseClusterManager.RemoteShell cannot override
getExecString() in org.apache.hadoop.util.Shell.ShellCommandExecutor;
attempting to assign weaker access privileges; was public
{noformat}
the issue is that getExecString() is declared public in hadoop-2.0, but
protected in hadoop-1.0. In HBase, it is declared protected, and you can't
downgrade from public to protected.
We can just declare the function public and that seems to work, but given that
in hadoop the class is declared
{code}
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
{code}
perhaps we should just copy the class into hbase source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6833) [WINDOWS] Java Milisecond precision on Windows
Enis Soztutar created HBASE-6833: Summary: [WINDOWS] Java Milisecond precision on Windows Key: HBASE-6833 URL: https://issues.apache.org/jira/browse/HBASE-6833 Project: HBase Issue Type: Sub-task Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar HBase relies on the system clock obtained by System.currentTimeMilis() to supply the version, if it is not provided by the client in Put's and Delete's. At HBASE-6832 and HBASE-6826, we discovered that on some platforms, the milis clock might not be updated every milisecond, which might cause unexpected behavior. We also did some preliminary analysis there. In this issue, we can further discuss and inspect whether it is a problem for main target platforms and if so what can be done. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6832) [WINDOWS] TestRegionObserverBypass failures
[
https://issues.apache.org/jira/browse/HBASE-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458339#comment-13458339
]
Enis Soztutar commented on HBASE-6832:
--
The root cause of the problem seems related to HBASE-6826, mainly the Put after
the Delete happens at the same milisecond, thus the Delete eclipses the
successive Put.
On linux, in my MBP, 5-10 ms passes between tests in testMulti()
{code}
1346122558760
1346122558784
1346122558798
1346122558807
1346122558816
{code}
But on windows, without sleeps, it is:
{code}
1346122747062
1346122747076
1346122747076
{code}
It might be the case that it is running faster on windows, or, the Java
implementation cannot provide enough precision unless Thread.sleep() is called.
There is actually a way to inject custom timing into HBase, implemented in
EnrivonmentEdge.java and EnvironmentEdgeManager.java. I'll try to see whether
using smt like IncrementingEnvironmentEdge makes more sense here.
Here is an interesting study on time measurements in java across platforms and
versions:
http://code.google.com/p/javasimon/wiki/SystemTimersGranularity
>From that study, it seems that Windows server 2008 should have 1 ms update
>frequency for the System.currentTimeMilis() counter, which is good news. In my
>own (non-scientific) testing though, I found my Windows Server 2008 R2 to
>update the ms counter less frequently, although nano time counter seems to do
>the job. But the test was run on a virtual Windows installation, so take these
>with a grain of salt.
||trial||nanos_diff||milis_diff||
|1|21617442|16|
|2|7798401|0|
|3|6879142|16|
HBase uses System.currentTimeMilis() to obtain a notion of the global clock. We
cannot use System.nanoTime() since it is not connected to the wall-clock, but
only used for measuring time intervals.
> [WINDOWS] TestRegionObserverBypass failures
> ---
>
> Key: HBASE-6832
> URL: https://issues.apache.org/jira/browse/HBASE-6832
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Labels: windows
>
> TestRegionObserverBypass.testMulti() fails with
> {code}
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:472)
> at org.junit.Assert.assertEquals(Assert.java:456)
> at
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass.checkRowAndDelete(TestRegionObserverBypass.java:173)
> at
> org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass.testMulti(TestRegionObserverBypass.java:166)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6832) [WINDOWS] TestRegionObserverBypass failures
Enis Soztutar created HBASE-6832:
Summary: [WINDOWS] TestRegionObserverBypass failures
Key: HBASE-6832
URL: https://issues.apache.org/jira/browse/HBASE-6832
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
TestRegionObserverBypass.testMulti() fails with
{code}
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass.checkRowAndDelete(TestRegionObserverBypass.java:173)
at
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass.testMulti(TestRegionObserverBypass.java:166)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6800) Build a Document Store on HBase for Better Query Processing
[ https://issues.apache.org/jira/browse/HBASE-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458331#comment-13458331 ] Jason Dai commented on HBASE-6800: -- bq. Considering that the relative size of individual field(s) in the document can be small, the cost of update would be comparatively higher than a fully de-normalized schema. One option is to take the similar approach as HBaseHUT (https://github.com/sematext/HBaseHUT), which converts RMW to updates, and constructs the up-to-date value on the fly. > Build a Document Store on HBase for Better Query Processing > --- > > Key: HBASE-6800 > URL: https://issues.apache.org/jira/browse/HBASE-6800 > Project: HBase > Issue Type: New Feature > Components: coprocessors, performance >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: dot-deisgn.pdf > > > In the last couple of years, increasingly more people begin to stream data > into HBase in near time, and > use high level queries (e.g., Hive) to analyze the data in HBase directly. > While HBase already has very effective MapReduce integration with its good > scanning performance, query processing using MapReduce on HBase still has > significant gaps compared to HDFS: ~3x space overheads and 3~5x performance > overheads according to our measurement. > We propose to implement a document store on HBase, which can greatly improve > query processing on HBase (by leveraging the relational model and read-mostly > access patterns). According to our prototype, it can reduce space usage by > up-to ~3x and speedup query processing by up-to ~1.8x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6831) [WINDOWS] HBaseTestingUtility.expireSession() does not expire zookeeper session
[
https://issues.apache.org/jira/browse/HBASE-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458329#comment-13458329
]
Enis Soztutar commented on HBASE-6831:
--
Upon further inspection the root cause boils down to the following sequence of
events.
HBaseTestingUtility.expireSession() works by getting an already existing
Zookeeper object, and obtains sessionId from that.
Then it constructs another Zookeeper() object with the same sessionId and
passwd, and closes that Zookeeper, which in turn triggers server to mark the
sessionId as expired.
{code}
ZooKeeper zk = nodeZK.getRecoverableZooKeeper().getZooKeeper();
byte[] password = zk.getSessionPasswd();
long sessionID = zk.getSessionId();
ZooKeeper newZK = new ZooKeeper(quorumServers,
sessionTimeout, EmptyWatcher.instance, sessionID, password);
newZK.close();
{code}
The race condition occurs, because Zookeeper() constructor starts the
ClientCnxn.SendThread, but does not wait for connection establishment.
SendThread.run() does connect to the server if it is not on CLOSING state, and
changes the state to be CONNECTING. However, Zookeeper.close() sets the state
as CLOSING, sends a closeSession event to server. Thus if we construct
Zookeeper() and immediately close it as above, then SendThread.run() and
ClientCnxn.close() races, and if close() is run first, then we abort the
connection, and do not even try to establish a connection.
On Windows, the new thread's execution does not seem to start as fast as linux,
so we did not run into this previously.
> [WINDOWS] HBaseTestingUtility.expireSession() does not expire zookeeper
> session
> ---
>
> Key: HBASE-6831
> URL: https://issues.apache.org/jira/browse/HBASE-6831
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Labels: windows
>
> TestReplicationPeer fails because it forces the zookeeper session expiration
> by calling HBaseTestingUtilty.expireSesssion(), but that function fails to do
> so.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6831) [WINDOWS] HBaseTestingUtility.expireSession() does not expire zookeeper session
Enis Soztutar created HBASE-6831: Summary: [WINDOWS] HBaseTestingUtility.expireSession() does not expire zookeeper session Key: HBASE-6831 URL: https://issues.apache.org/jira/browse/HBASE-6831 Project: HBase Issue Type: Bug Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar TestReplicationPeer fails because it forces the zookeeper session expiration by calling HBaseTestingUtilty.expireSesssion(), but that function fails to do so. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6830) [WINDOWS] Tests should not rely on local temp dir to be available in DFS
Enis Soztutar created HBASE-6830:
Summary: [WINDOWS] Tests should not rely on local temp dir to be
available in DFS
Key: HBASE-6830
URL: https://issues.apache.org/jira/browse/HBASE-6830
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Some of the tests resolve the local temp directory for temporary test data, but
use this directory path in dfs. Since on windows, local temp dir is resolved to
something like: c:\\, DistributedFileSystem.getPathName()
throws an IllegalArgumentException complaining that it is not a valid path
name.
Instead of relying on a local temp dir name, we should create a temp dir on
dfs, and use this as a basis dir for test data.
At least the following test cases are affected by this:
{code}
TestHFileOutputFormat
TestHRegionServerBulkLoad
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6829) [WINDOWS] TestCacheOnWriteInSchema failures
Enis Soztutar created HBASE-6829:
Summary: [WINDOWS] TestCacheOnWriteInSchema failures
Key: HBASE-6829
URL: https://issues.apache.org/jira/browse/HBASE-6829
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
TestCacheOnWriteInSchema fails with
{code}
java.io.IOException: Target HLog directory already exists:
./target/test-data/2d814e66-75d3-4c1b-92c7-a49d9972e8fd/TestCacheOnWriteInSchema/logs
at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:385)
at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:316)
at
org.apache.hadoop.hbase.regionserver.TestCacheOnWriteInSchema.setUp(TestCacheOnWriteInSchema.java:162)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6829) [WINDOWS] TestCacheOnWriteInSchema failures
[
https://issues.apache.org/jira/browse/HBASE-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458327#comment-13458327
]
Enis Soztutar commented on HBASE-6829:
--
The problem is that HLog is not closed between tests, which causes the deletion
of the log directory to fail.
> [WINDOWS] TestCacheOnWriteInSchema failures
> ---
>
> Key: HBASE-6829
> URL: https://issues.apache.org/jira/browse/HBASE-6829
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Labels: windows
>
> TestCacheOnWriteInSchema fails with
> {code}
> java.io.IOException: Target HLog directory already exists:
> ./target/test-data/2d814e66-75d3-4c1b-92c7-a49d9972e8fd/TestCacheOnWriteInSchema/logs
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:385)
> at org.apache.hadoop.hbase.regionserver.wal.HLog.(HLog.java:316)
> at
> org.apache.hadoop.hbase.regionserver.TestCacheOnWriteInSchema.setUp(TestCacheOnWriteInSchema.java:162)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6827) [WINDOWS] TestScannerTimeout fails expecting a timeout
[
https://issues.apache.org/jira/browse/HBASE-6827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458322#comment-13458322
]
Enis Soztutar commented on HBASE-6827:
--
The failure seems to be related to timing. We wait for SCANNER_TIMEOUT
(10secs), but the Leases.java only wakes up once in
HConstants.THREAD_WAKE_FREQUENCY (10 sec by default, 1 sec in tests). But the
wait should actually ensure that we wait for at least
SCANNER_TIMEOUT + THREAD_WAKE_FREQUENCY.
> [WINDOWS] TestScannerTimeout fails expecting a timeout
> --
>
> Key: HBASE-6827
> URL: https://issues.apache.org/jira/browse/HBASE-6827
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> TestScannerTimeout.test2481() fails with:
> {code}
> java.lang.AssertionError: We should be timing out
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.hadoop.hbase.client.TestScannerTimeout.test2481(TestScannerTimeout.java:117)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6828) [WINDOWS] TestMemoryBoundedLogMessageBuffer failures
Enis Soztutar created HBASE-6828: Summary: [WINDOWS] TestMemoryBoundedLogMessageBuffer failures Key: HBASE-6828 URL: https://issues.apache.org/jira/browse/HBASE-6828 Project: HBase Issue Type: Bug Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar TestMemoryBoundedLogMessageBuffer fails because of a suspected \n line ending difference. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6827) [WINDOWS] TestScannerTimeout fails expecting a timeout
Enis Soztutar created HBASE-6827:
Summary: [WINDOWS] TestScannerTimeout fails expecting a timeout
Key: HBASE-6827
URL: https://issues.apache.org/jira/browse/HBASE-6827
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
TestScannerTimeout.test2481() fails with:
{code}
java.lang.AssertionError: We should be timing out
at org.junit.Assert.fail(Assert.java:93)
at
org.apache.hadoop.hbase.client.TestScannerTimeout.test2481(TestScannerTimeout.java:117)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6826) [WINDOWS] TestFromClientSide failures
Enis Soztutar created HBASE-6826:
Summary: [WINDOWS] TestFromClientSide failures
Key: HBASE-6826
URL: https://issues.apache.org/jira/browse/HBASE-6826
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
The following tests fail for TestFromClientSide:
{code}
testPoolBehavior()
testClientPoolRoundRobin()
testClientPoolThreadLocal()
{code}
The first test fails due to the fact that the test (wrongly) assumes that
ThredPoolExecutor can reclaim the thread immediately.
The second and third tests seem to fail because that Put's to the table does
not specify an explicit timestamp, but on windows, consecutive calls to put
happen to finish in the same milisecond so that the resulting mutations have
the same timestamp, thus there is only one version of the cell value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6825) [WINDOWS] Java NIO socket channels does not work with Windows ipv6
[
https://issues.apache.org/jira/browse/HBASE-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458315#comment-13458315
]
Enis Soztutar commented on HBASE-6825:
--
The ipv6 test attached at http://bugs.sun.com/view_bug.do?bug_id=6230761 works
with jdk1.7.0_6, but not with jdk1.6.0_33 (Windows Server 2008 R2).
{code}
PS C:\Program Files\Java\jdk1.6.0_33\bin> .\java.exe -cp '.\target\classes' ipv6
==> 1
==> 2
java.net.SocketException: Address family not supported by protocol family: bind
at sun.nio.ch.Net.bind(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:52)
at ipv6.main(ipv6.java:30)
{code}
{code}
PS C:\Program Files\Java\jdk1.7.0_06\bin> .\java.exe -cp '.\target\classes' ipv6
==> 1
==> 2
==> 3
{code}
There are 3 possible workarounds:
1.) set socket address preference to IPV4 –
{code}
-Djava.net.preferIPv4Stack=true
{code}
2.) use "classic" sockets instead of NIO:
{code}
ServerSocket s = new ServerSocket();
s.bind(new InetSocketAddress(InetAddress.getByName("::"), 0));
{code}
3) set
{code}
reg add
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TCPIP6\Parameters\ /v
DisabledComponents /t REG_BINARY /d 20
{code}
setting it either 0x20 of 0x.
http://support.microsoft.com/kb/929852
http://serverfault.com/questions/218745/disable-ipv6-on-loopback-address-localhost-computer-name
This will either disable ipv6, or cause preference of v4 over v6 on the node.
However, It seems that 1) is the best approach right now. Note that we have to
set -Djava.net.preferIPv4Stack=true at maven (for tests), hbase-env (for
deployment), and eclipse (if used for development). I'll provide a patch.
Following shows that preferIPv4 works as intended:
{code}
public class TestPreferIpv4 {
@Test
public void testPreferIpv4() throws Exception {
System.setProperty("java.net.preferIPv4Stack" , "true");
InetAddress[] addrs = InetAddress.getAllByName("localhost");
for (InetAddress addr : addrs) {
System.out.println(addr);
}
}
public static void main(String[] args) throws Exception {
new TestPreferIpv4().testPreferIpv4();
}
}
{code}
> [WINDOWS] Java NIO socket channels does not work with Windows ipv6
> --
>
> Key: HBASE-6825
> URL: https://issues.apache.org/jira/browse/HBASE-6825
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.96.0, 0.94.3
> Environment: JDK6 on windows for ipv6.
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> While running the test TestAdmin.testCheckHBaseAvailableClosesConnection(), I
> noticed that it takes very long, since it sleeps for 2sec * 500, because of
> zookeeper retries.
> The root cause of the problem is that ZK uses Java NIO to create
> ServerSorcket's from ServerSocketChannels. Under windows, the ipv4 and ipv6
> is implemented independently, and Java seems that it cannot reuse the same
> socket channel for both ipv4 and ipv6 sockets. We are getting
> "java.net.SocketException: Address family not supported by protocol
> family" exceptions. When, ZK client resolves "localhost", it gets both v4
> 127.0.0.1 and v6 ::1 address, but the socket channel cannot bind to both v4
> and v6.
> The problem is reported as:
> http://bugs.sun.com/view_bug.do?bug_id=6230761
> http://stackoverflow.com/questions/1357091/binding-an-ipv6-server-socket-on-windows
> Although the JDK bug is reported as resolved, I have tested with jdk1.6.0_33
> without any success. Although JDK7 seems to have fixed this problem. In ZK,
> we can replace the ClientCnxnSocket implementation from ClientCnxnSocketNIO
> to a non-NIO one, but I am not sure that would be the way to go.
> Disabling ipv6 resolution of "localhost" is one other approach. I'll test it
> to see whether it will be any good.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6825) [WINDOWS] Java NIO socket channels does not work with Windows ipv6
Enis Soztutar created HBASE-6825: Summary: [WINDOWS] Java NIO socket channels does not work with Windows ipv6 Key: HBASE-6825 URL: https://issues.apache.org/jira/browse/HBASE-6825 Project: HBase Issue Type: Sub-task Affects Versions: 0.96.0, 0.94.3 Environment: JDK6 on windows for ipv6. Reporter: Enis Soztutar Assignee: Enis Soztutar While running the test TestAdmin.testCheckHBaseAvailableClosesConnection(), I noticed that it takes very long, since it sleeps for 2sec * 500, because of zookeeper retries. The root cause of the problem is that ZK uses Java NIO to create ServerSorcket's from ServerSocketChannels. Under windows, the ipv4 and ipv6 is implemented independently, and Java seems that it cannot reuse the same socket channel for both ipv4 and ipv6 sockets. We are getting "java.net.SocketException: Address family not supported by protocol family" exceptions. When, ZK client resolves "localhost", it gets both v4 127.0.0.1 and v6 ::1 address, but the socket channel cannot bind to both v4 and v6. The problem is reported as: http://bugs.sun.com/view_bug.do?bug_id=6230761 http://stackoverflow.com/questions/1357091/binding-an-ipv6-server-socket-on-windows Although the JDK bug is reported as resolved, I have tested with jdk1.6.0_33 without any success. Although JDK7 seems to have fixed this problem. In ZK, we can replace the ClientCnxnSocket implementation from ClientCnxnSocketNIO to a non-NIO one, but I am not sure that would be the way to go. Disabling ipv6 resolution of "localhost" is one other approach. I'll test it to see whether it will be any good. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6824) [WINDOWS] TestClassLoading.testClassLoadingFromHDFS() fails
Enis Soztutar created HBASE-6824:
Summary: [WINDOWS] TestClassLoading.testClassLoadingFromHDFS()
fails
Key: HBASE-6824
URL: https://issues.apache.org/jira/browse/HBASE-6824
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Two HBase TestClassLoading unit tests failed due to a failiure in loading the
test file from HDFS:
{code}
testClassLoadingFromHDFS(org.apache.hadoop.hbase.coprocessor.TestClassLoading):
Class TestCP1 was missing on a region
testClassLoadingFromLibDirInJar(org.apache.hadoop.hbase.coprocessor.TestClassLoading):
Class TestCP1 was missing on a region
{code}
The problem is that CoprocessorHost.load() copies the jar file locally, and
schedules the local file to be deleted on exit, but calling
FileSystem.deleteOnExit(). However, the filesystem is not the file system of
the local file, it is the distributed file system, so on windows, the Path
fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6823) [WINDOWS] TestSplitTransaction fails due the Log handle is not released by a call to the DaughterOpener.start()
Enis Soztutar created HBASE-6823: Summary: [WINDOWS] TestSplitTransaction fails due the Log handle is not released by a call to the DaughterOpener.start() Key: HBASE-6823 URL: https://issues.apache.org/jira/browse/HBASE-6823 Project: HBase Issue Type: Bug Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar There are two unit test cases in HBase RegionServer test failed in the clean up stage that failed to delete the files/folders created in the test. testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): Failed delete of ./target/test- data/1c386abc-f159-492e-b21f-e89fab24d85b/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/table/a588d813fd26280c2b42e93565ed960c testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): Failed delete of ./target/test-data/6 1a1a14b-0cc9-4dd6-93fd-4dc021e2bfcc/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/table/8090abc89528461fa284288c257662cd The root cause is triggered by ta call to the DaughterOpener.start() in \src\hbase\src\main\java\org\apache\hadoop\hbase\regionserver\SplitTransactopn.Java (openDaughters() function). It left handles to the splited folder/file and causing deleting of the file/folder failed in the Windows OS. Windows does not allow to delete a file, while there are open file handlers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6822) [WINDOWS] MiniZookeeperCluster multiple daemons bind to the same port
Enis Soztutar created HBASE-6822:
Summary: [WINDOWS] MiniZookeeperCluster multiple daemons bind to
the same port
Key: HBASE-6822
URL: https://issues.apache.org/jira/browse/HBASE-6822
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
TestHBaseTestingUtility.testMiniZooKeeper() tests whether the mini zk cluster
is working by launching 5 threads corresponding to zk servers.
NIOServerCnxnFactory.configure() configures the socket as:
{code}
this.ss = ServerSocketChannel.open();
ss.socket().setReuseAddress(true);
{code}
setReuseAddress() is set, because it allows the server to come back up and bind
to the same port before the socket is timed-out by the kernel.
Under windows, the behavior on ServerSocket.setReuseAddress() is different than
on linux, in which it allows any process to bind to an already-bound port. This
causes ZK nodes starting on the same node, to be able to bind to the same port.
The following part of the patch at
https://issues.apache.org/jira/browse/HADOOP-8223 deals with this case for
Hadoop:
{code}
if(Shell.WINDOWS) {
+ // result of setting the SO_REUSEADDR flag is different on Windows
+ // http://msdn.microsoft.com/en-us/library/ms740621(v=vs.85).aspx
+ // without this 2 NN's can start on the same machine and listen on
+ // the same port with indeterminate routing of incoming requests to them
+ ret.setReuseAddress(false);
+}
{code}
We should do the same in Zookeeper (I'll open a ZOOK issue). But in the
meantime, we can fix hbase tests to not rely on BindException to resolve for
bind errors. Especially, in MiniZKCluster.startup() when starting more than 1
servers, we already know that we have to increment the port number.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6821) [WINDOWS] .META. table name causes file system problems in windows
Enis Soztutar created HBASE-6821: Summary: [WINDOWS] .META. table name causes file system problems in windows Key: HBASE-6821 URL: https://issues.apache.org/jira/browse/HBASE-6821 Project: HBase Issue Type: Bug Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar TestMetaMigrationRemovingHTD untars a cluster dir having a .META. subdirectory. This causes mvn clean to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6649: --- Attachment: 6649-fix-io-exception-handling.patch This patch demonstrates what I commented with earlier. Please have a look. I could make a method which has the getPosition() and next().. but I wanted to check on whether folks agree with the fix first. > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-fix-io-exception-handling.patch, 6649-trunk.patch, 6649-trunk.patch, > 6649.txt, HBase-0.92 #495 test - queueFailover [Jenkins].html, HBase-0.92 > #502 test - queueFailover [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6820) [WINDOWS] MiniZookeeperCluster should ensure that ZKDatabase is closed upon shutdown()
Enis Soztutar created HBASE-6820:
Summary: [WINDOWS] MiniZookeeperCluster should ensure that
ZKDatabase is closed upon shutdown()
Key: HBASE-6820
URL: https://issues.apache.org/jira/browse/HBASE-6820
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.3
Reporter: Enis Soztutar
Assignee: Enis Soztutar
MiniZookeeperCluster.shutdown() shuts down the ZookeeperServer and
NIOServerCnxnFactory. However, MiniZookeeperCluster uses a deprecated
ZookeeperServer constructor, which in turn constructs its own FileTxnSnapLog,
and ZKDatabase. Since ZookeeperServer.shutdown() does not close() the
ZKDatabase, we have to explicitly close it in MiniZookeeperCluster.shutdown().
Tests effected by this are
{code}
TestSplitLogManager
TestSplitLogWorker
TestOfflineMetaRebuildBase
TestOfflineMetaRebuildHole
TestOfflineMetaRebuildOverlap
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6779) Fix issues analysis.apache.org raises about StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458292#comment-13458292 ] Hudson commented on HBASE-6779: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6779 Fix issues analysis.apache.org raises about StochasticLoadBalancer (Revision 1387372) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java > Fix issues analysis.apache.org raises about StochasticLoadBalancer > -- > > Key: HBASE-6779 > URL: https://issues.apache.org/jira/browse/HBASE-6779 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: HBASE-6779-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6814) [WINDOWS] HBase on Windows
[ https://issues.apache.org/jira/browse/HBASE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458293#comment-13458293 ] Enis Soztutar commented on HBASE-6814: -- We should probably agree on a plan for how to merge these work into trunk. We can mark the windows-specific issues with [WINDOWS] prefix so that they are easily searchable. We can go with either of these: (1) recently agreed github repo approach (2) create a branch on apache svn, start committing there, and merge when done (3) commit patches to trunk as they are reviewed. I am up for (1), and (3). The only downside of (1) is that there might be multiple people involved, I am up for any suggestions that would make reviews more easy (not trying to recap the dev mailing list discussion here). > [WINDOWS] HBase on Windows > -- > > Key: HBASE-6814 > URL: https://issues.apache.org/jira/browse/HBASE-6814 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > This is an umbrella jira to support windows as a first class citizen. > The short term goals are: > - Get unit tests passing > - Scripts converted to windows > - Runtime does not depend on cgywin (build can still depend on cygwin for > now) > - Hbase on windows will consume hadoop branch-1-win artifacts. > - Get nightly jenkins build on windows -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6412) Move external servers to metrics2 (thrift,thrift2,rest)
[ https://issues.apache.org/jira/browse/HBASE-6412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458289#comment-13458289 ] Hudson commented on HBASE-6412: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6412 Move external servers to metrics2 (thrift,thrift2,rest) (Revision 1387323) Result = FAILURE stack : Files : * /hbase/trunk/dev-support/test-patch.sh * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/CompatibilityFactory.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/CompatibilitySingletonFactory.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/rest * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/rest/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/thrift * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/thrift/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/thrift/metrics/ThriftServerMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/thrift/metrics/ThriftServerMetricsSourceFactory.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSourceFactoryTest.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterMetricsSourceFactory.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceFactoryTest.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics/TestReplicationMetricsSourceFactory.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/rest * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/rest/metrics * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/rest/metrics/TestRESTMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/test * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/test/MetricsAssertHelper.java * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/thrift * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/thrift/metrics * /hbase/trunk/hbase-hadoop-compat/src/test/java/org/apache/hadoop/hbase/thrift/metrics/TestThriftServerMetricsSourceFactory.java * /hbase/trunk/hbase-hadoop1-compat/pom.xml * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSourceFactoryImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/rest * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/rest/metrics * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/rest/metrics/RESTMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/thrift * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/thrift/metrics * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/thrift/metrics/ThriftServerMetricsSourceFactoryImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/thrift/metrics/ThriftServerMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.rest.metrics.RESTMetricsSource * /hbase/trunk/hbase-hadoop1-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.thrift.metrics.ThriftServerMetricsSourceFactory * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/metrics/TestBaseMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop1-compat/src/test/java/org/apache/hadoop/hbase/replication/regionserver/metrics/ReplicationMetricsSourceImplTest.java * /hbase/trunk/hbase-hadoop1-compat/
[jira] [Commented] (HBASE-6809) Deprecate Old metrics classes.
[ https://issues.apache.org/jira/browse/HBASE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458290#comment-13458290 ] Hudson commented on HBASE-6809: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6809 Deprecate Old metrics classes. (Revision 1387359) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/ExactCounterMetric.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/HBaseInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/MetricsMBeanBase.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/MetricsRate.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/MetricsString.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/PersistentMetricsTimeVaryingRate.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/file/TimeStampingFileContext.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/metrics/histogram/MetricsHistogram.java > Deprecate Old metrics classes. > -- > > Key: HBASE-6809 > URL: https://issues.apache.org/jira/browse/HBASE-6809 > Project: HBase > Issue Type: Sub-task > Components: metrics >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 0.96.0 > > Attachments: HBASE-6809-trunk-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6795) mvn compile fails on a fresh checkout with empty ~/.m2/repo
[
https://issues.apache.org/jira/browse/HBASE-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458291#comment-13458291
]
Hudson commented on HBASE-6795:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/])
HBASE-6795 mvn compile fails on a fresh checkout with empty ~/.m2/repo
(Revision 1387338)
Result = FAILURE
stack :
Files :
* /hbase/trunk/pom.xml
> mvn compile fails on a fresh checkout with empty ~/.m2/repo
> ---
>
> Key: HBASE-6795
> URL: https://issues.apache.org/jira/browse/HBASE-6795
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 0.96.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: 6795.txt
>
>
> I have noticed that mvn compile fails if your ~/m2/repository/ does not
> contain hbase test jars, however mvn test-compile, mvn install, etc works as
> expected.
> The patch for HBASE-6706 introduced test-jar dependency from hbase-server and
> hbase-hadoop1-compat to hbase-hadoop-compat test jar in the test scope. But
> stupid maven still tries to resolve the test jar when you do maven compile
> (notice that we are not even in the test scope).
> mvn test-compile, etc works b/c the test-jar for hbase-hadoop-compat is build
> before hbase-hadoop1-compat.
> One way to solve this is to push SNAPSHOT test-jars for hbase-hadoop-compat
> to the snapshot repository, so next time, they are referenced from there.
> Other alternative is to move classes under hbase-hadoop{|1|2}-compat/src/test
> to src/main, and remove the test-jar intra-module dependency. Still, it seems
> we might need intra-module test-jar dependency in the future.
> Any other suggestions are welcome.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6438) RegionAlreadyInTransitionException needs to give more info to avoid assignment inconsistencies
[ https://issues.apache.org/jira/browse/HBASE-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458288#comment-13458288 ] Hudson commented on HBASE-6438: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6438 Addendum checks regionAlreadyInTransitionException when generating region plan (Chunhui) (Revision 1387164) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > RegionAlreadyInTransitionException needs to give more info to avoid > assignment inconsistencies > -- > > Key: HBASE-6438 > URL: https://issues.apache.org/jira/browse/HBASE-6438 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: rajeshbabu > Fix For: 0.96.0, 0.92.3, 0.94.3 > > Attachments: 6438-0.92.txt, 6438.addendum, 6438-addendum.94, > 6438-trunk_2.patch, HBASE-6438_2.patch, HBASE-6438_94_3.patch, > HBASE-6438_94_4.patch, HBASE-6438_94.patch, HBASE-6438-trunk_2.patch, > HBASE-6438_trunk.patch > > > Seeing some of the recent issues in region assignment, > RegionAlreadyInTransitionException is one reason after which the region > assignment may or may not happen(in the sense we need to wait for the TM to > assign). > In HBASE-6317 we got one problem due to RegionAlreadyInTransitionException on > master restart. > Consider the following case, due to some reason like master restart or > external assign call, we try to assign a region that is already getting > opened in a RS. > Now the next call to assign has already changed the state of the znode and so > the current assign that is going on the RS is affected and it fails. The > second assignment that started also fails getting RAITE exception. Finally > both assignments not carrying on. Idea is to find whether any such RAITE > exception can be retried or not. > Here again we have following cases like where > -> The znode is yet to transitioned from OFFLINE to OPENING in RS > -> RS may be in the step of openRegion. > -> RS may be trying to transition OPENING to OPENED. > -> RS is yet to add to online regions in the RS side. > Here in openRegion() and updateMeta() any failures we are moving the znode to > FAILED_OPEN. So in these cases getting an RAITE should be ok. But in other > cases the assignment is stopped. > The idea is to just add the current state of the region assignment in the RIT > map in the RS side and using that info we can determine whether the > assignment can be retried or not on getting an RAITE. > Considering the current work going on in AM, pls do share if this is needed > atleast in the 0.92/0.94 versions? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6409) Create histogram class for metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458285#comment-13458285 ] Hudson commented on HBASE-6409: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6409 Create histogram class for metrics 2 (Revision 1387358) Result = FAILURE stack : Files : * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSource.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics/MetricHistogram.java * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/metrics/MetricsExecutor.java * /hbase/trunk/hbase-hadoop1-compat/pom.xml * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/lib/MetricMutableHistogram.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/lib/MetricMutableQuantiles.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/lib/MetricsExecutorImpl.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/util * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/util/MetricQuantile.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/metrics2/util/MetricSampleQuantiles.java * /hbase/trunk/hbase-hadoop2-compat/pom.xml * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MetricMutableQuantiles.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MetricsExecutorImpl.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableHistogram.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/util * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/util/MetricQuantile.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/util/MetricSampleQuantiles.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java > Create histogram class for metrics 2 > > > Key: HBASE-6409 > URL: https://issues.apache.org/jira/browse/HBASE-6409 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 0.96.0 > > Attachments: 6409v6.txt, HBASE-6409-0.patch, HBASE-6409-1.patch, > HBASE-6409-2.patch, HBASE-6409-3.patch, HBASE-6409-4.patch, > HBASE-6409-5.patch, HBASE-6409-6.patch > > > Create the replacement for MetricsHistogram and PersistantTimeVaryingRate > classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6803) script hbase should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
[ https://issues.apache.org/jira/browse/HBASE-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458286#comment-13458286 ] Hudson commented on HBASE-6803: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6803 script hbase should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH (Revision 1387258) Result = FAILURE jxiang : Files : * /hbase/trunk/bin/hbase > script hbase should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH > > > Key: HBASE-6803 > URL: https://issues.apache.org/jira/browse/HBASE-6803 > Project: HBase > Issue Type: Bug > Components: shell >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0, 0.94.2 > > Attachments: trunk-6803.patch > > > Snappy SO fails to load properly if LD_LIBRARY_PATH does not include the path > where snappy SO is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458287#comment-13458287 ] Hudson commented on HBASE-6592: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #180 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/180/]) HBASE-6592 [shell] Add means of custom formatting output by column (Revision 1387369) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/get.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/scan.rb * /hbase/trunk/hbase-server/src/test/ruby/hbase/table_test.rb > [shell] Add means of custom formatting output by column > --- > > Key: HBASE-6592 > URL: https://issues.apache.org/jira/browse/HBASE-6592 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: stack >Assignee: Jie Huang >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: 6592v3.txt, hbase-6592.patch, hbase-6592-v2.patch, > hbase-6952-v1.patch > > > See Jacques suggestion toward end of this thread for how we should allow > adding a custom formatter per column to use outputting column content in > shell: > http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shell&subj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6817) [WINDOWS] Get HBase tests working under Windows
[
https://issues.apache.org/jira/browse/HBASE-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458279#comment-13458279
]
Enis Soztutar commented on HBASE-6817:
--
Here are the failing tests:
{code}
Running org.apache.hadoop.hbase.mapred.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 62.892 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 111.315 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
Tests run: 6, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 24.857 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery
Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 36.475 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 65.058 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 50.192 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 58.588 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestTimeRangeMapRed
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 45.262 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestWALPlayer
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 40.744 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Tests run: 4, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 348.05 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 58.316 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 614.667 sec
<<< FAILURE!
Running org.apache.hadoop.hbase.replication.TestMasterReplication
Tests run: 2, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 692.191 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 331.081 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.replication.TestReplication
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 495.861 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.TestRegionRebalancing
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 564.325 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.TestZooKeeper
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 536.644 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 422.381 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 23.788 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.coprocessor.TestClassLoading
Tests run: 7, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 68.36 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 68.008 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestImportExport
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 46.673 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.mapreduce.TestImportTsv
Tests run: 9, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 88.915 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.master.TestSplitLogManager
Tests run: 12, Failures: 0, Errors: 9, Skipped: 0, Time elapsed: 14.718 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.monitoring.TestMemoryBoundedLogMessageBuffer
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.334 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.regionserver.TestCacheOnWriteInSchema
Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.747 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.318 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.097 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 39.134 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 36.861 sec <<<
FAILURE!
Running org.apache.hadoop.hbase.util.TestFSUtils
Tests run: 4, Failures: 0, Errors: 1, Skipped
[jira] [Updated] (HBASE-6817) [WINDOWS] Get HBase tests working under Windows
[
https://issues.apache.org/jira/browse/HBASE-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-6817:
-
Attachment: hbase-trunk-runLargeTests.txt
hbase-trunk-runMediumTests.txt
hbase-trunk-runSmallTests.txt
hbase-0.94-runLargeTests.txt
hbase-0.94-runMediumTests.txt
hbase-0.94-runSmallTests.txt
Attaching tests results from a fresh checkout for 0.94 and trunk without any
windows-specific patches in.
The summary for 0.94 is:
{code}
small:
Tests run: 567, Failures: 0, Errors: 3, Skipped: 0
medium:
Tests run: 651, Failures: 3, Errors: 24, Skipped: 2
large:
Tests run: 295, Failures: 1, Errors: 31, Skipped: 9
{code}
For trunk, summary for the small/medium/large tests from hbase-server are
below. hbase-common and other modules' tests run fine.
{code}
Tests run: 586, Failures: 1, Errors: 3, Skipped: 0
Tests run: 698, Failures: 12, Errors: 20, Skipped: 2
Tests run: 369, Failures: 6, Errors: 45, Skipped: 9
{code}
> [WINDOWS] Get HBase tests working under Windows
> ---
>
> Key: HBASE-6817
> URL: https://issues.apache.org/jira/browse/HBASE-6817
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Affects Versions: 0.96.0, 0.94.3
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Labels: windows
> Attachments: hbase-0.94-runLargeTests.txt,
> hbase-0.94-runMediumTests.txt, hbase-0.94-runSmallTests.txt,
> hbase-trunk-runLargeTests.txt, hbase-trunk-runMediumTests.txt,
> hbase-trunk-runSmallTests.txt
>
>
> This is an umbrella jira for getting HBase unit tests working under windows.
> We can link to individual test fixes, and track the general progress here.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6819) [WINDOWS] Setup Jenkins CI for HBase on Windows
Enis Soztutar created HBASE-6819: Summary: [WINDOWS] Setup Jenkins CI for HBase on Windows Key: HBASE-6819 URL: https://issues.apache.org/jira/browse/HBASE-6819 Project: HBase Issue Type: Sub-task Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar Once we get tests working, we should setup a Jenkins build. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458274#comment-13458274 ] Devaraj Das commented on HBASE-6649: Okay a plausible explanation - 1. ReplicationSource.readAllEntriesToReplicateOrNextFile throws an IOException (which causes the log "Break on IOE:" to print), but ignores the exception. 2. When readAllEntriesToReplicateOrNextFile returns, the reader's file-pointer position is queried and 'this.position' is set to that (the reader's file-pointer is possibly pointing to gibberish) 3. Eventually, readAllEntriesToReplicateOrNextFile gets called again, and this time this.reader.next inside throws IndexOutOfBounds exception because it read gibberish (looking at the code of DataInputStream.java, it seems like one of the cases where the IndexOutOfBounds is thrown is when the length passed to readFully is less than 0). The fix I can think of is to reset the reader's 'position' to the last valid position (upon return from the method readAllEntriesToReplicateOrNextFile). Thoughts on the above? Does the analysis make sense? > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-trunk.patch, 6649-trunk.patch, 6649.txt, HBase-0.92 #495 test - > queueFailover [Jenkins].html, HBase-0.92 #502 test - queueFailover > [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6818) [WINDOWS] Catalog table .META. violates the naming convention in Window OS
[ https://issues.apache.org/jira/browse/HBASE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458273#comment-13458273 ] Enis Soztutar commented on HBASE-6818: -- This is a hard one to tackle, since the META table's name is all over the code base. It is not straight, or even sensible, to change the name of the META table. Instead, we can work on a solution that changes the file name of the META table at the file system level, if the underlying filesystem is local (and windows). we also have to standardize on the local file name of the META table, so that we dont allow user level tables of that name. > [WINDOWS] Catalog table .META. violates the naming convention in Window OS > -- > > Key: HBASE-6818 > URL: https://issues.apache.org/jira/browse/HBASE-6818 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > There are two catalog tables in HBase, ROOT and .META., whose representitives > in Windows file system are folders. However, the name of .META. table > violates the naming convention in Windows, as Windows doesn't support any > file/directory whose name ending with a period '.'. > The following are the related description from msdn.microsoft.com for naming > convention for Windows : > Do not end a file or directory name with a space or a period. Although the > underlying file system may support such names, the Windows shell and user > interface does not. However, it is acceptable to specify a period as the > first character of a name. For example, ".temp". > Note that this does not affect hdfs, but only RawLocalFileSystem, and > single-node local hbase clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6818) [WINDOWS] Catalog table .META. violates the naming convention in Window OS
[ https://issues.apache.org/jira/browse/HBASE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6818: - Labels: windows (was: ) > [WINDOWS] Catalog table .META. violates the naming convention in Window OS > -- > > Key: HBASE-6818 > URL: https://issues.apache.org/jira/browse/HBASE-6818 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > There are two catalog tables in HBase, ROOT and .META., whose representitives > in Windows file system are folders. However, the name of .META. table > violates the naming convention in Windows, as Windows doesn't support any > file/directory whose name ending with a period '.'. > The following are the related description from msdn.microsoft.com for naming > convention for Windows : > Do not end a file or directory name with a space or a period. Although the > underlying file system may support such names, the Windows shell and user > interface does not. However, it is acceptable to specify a period as the > first character of a name. For example, ".temp". > Note that this does not affect hdfs, but only RawLocalFileSystem, and > single-node local hbase clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6818) [WINDOWS] Catalog table .META. violates the naming convention in Window OS
Enis Soztutar created HBASE-6818: Summary: [WINDOWS] Catalog table .META. violates the naming convention in Window OS Key: HBASE-6818 URL: https://issues.apache.org/jira/browse/HBASE-6818 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar There are two catalog tables in HBase, ROOT and .META., whose representitives in Windows file system are folders. However, the name of .META. table violates the naming convention in Windows, as Windows doesn't support any file/directory whose name ending with a period '.'. The following are the related description from msdn.microsoft.com for naming convention for Windows : Do not end a file or directory name with a space or a period. Although the underlying file system may support such names, the Windows shell and user interface does not. However, it is acceptable to specify a period as the first character of a name. For example, ".temp". Note that this does not affect hdfs, but only RawLocalFileSystem, and single-node local hbase clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6818) [WINDOWS] Catalog table .META. violates the naming convention in Window OS
[ https://issues.apache.org/jira/browse/HBASE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6818: - Affects Version/s: 0.94.3 0.96.0 > [WINDOWS] Catalog table .META. violates the naming convention in Window OS > -- > > Key: HBASE-6818 > URL: https://issues.apache.org/jira/browse/HBASE-6818 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > There are two catalog tables in HBase, ROOT and .META., whose representitives > in Windows file system are folders. However, the name of .META. table > violates the naming convention in Windows, as Windows doesn't support any > file/directory whose name ending with a period '.'. > The following are the related description from msdn.microsoft.com for naming > convention for Windows : > Do not end a file or directory name with a space or a period. Although the > underlying file system may support such names, the Windows shell and user > interface does not. However, it is acceptable to specify a period as the > first character of a name. For example, ".temp". > Note that this does not affect hdfs, but only RawLocalFileSystem, and > single-node local hbase clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6818) [WINDOWS] Catalog table .META. violates the naming convention in Window OS
[ https://issues.apache.org/jira/browse/HBASE-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6818: - Assignee: Enis Soztutar > [WINDOWS] Catalog table .META. violates the naming convention in Window OS > -- > > Key: HBASE-6818 > URL: https://issues.apache.org/jira/browse/HBASE-6818 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: windows > > There are two catalog tables in HBase, ROOT and .META., whose representitives > in Windows file system are folders. However, the name of .META. table > violates the naming convention in Windows, as Windows doesn't support any > file/directory whose name ending with a period '.'. > The following are the related description from msdn.microsoft.com for naming > convention for Windows : > Do not end a file or directory name with a space or a period. Although the > underlying file system may support such names, the Windows shell and user > interface does not. However, it is acceptable to specify a period as the > first character of a name. For example, ".temp". > Note that this does not affect hdfs, but only RawLocalFileSystem, and > single-node local hbase clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6817) [WINDOWS] Get HBase tests working under Windows
Enis Soztutar created HBASE-6817: Summary: [WINDOWS] Get HBase tests working under Windows Key: HBASE-6817 URL: https://issues.apache.org/jira/browse/HBASE-6817 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar This is an umbrella jira for getting HBase unit tests working under windows. We can link to individual test fixes, and track the general progress here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6816) [WINDOWS] line endings on checkout for .sh files
Enis Soztutar created HBASE-6816: Summary: [WINDOWS] line endings on checkout for .sh files Key: HBASE-6816 URL: https://issues.apache.org/jira/browse/HBASE-6816 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Enis Soztutar On code checkout from svn or git, we need to ensure that the line endings for .sh files are LF, so that they work with cygwin. This is important for getting src/saveVersion.sh to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6815) [WINDOWS] Provide hbase scripts in order to start HBASE on Windows in a single user mode
[ https://issues.apache.org/jira/browse/HBASE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6815: - Issue Type: Sub-task (was: Improvement) Parent: HBASE-6814 > [WINDOWS] Provide hbase scripts in order to start HBASE on Windows in a > single user mode > > > Key: HBASE-6815 > URL: https://issues.apache.org/jira/browse/HBASE-6815 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.94.3 >Reporter: Enis Soztutar > > Provide .cmd scripts in order to start HBASE on Windows in a single user mode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6815) [WINDOWS] Provide hbase scripts in order to start HBASE on Windows in a single user mode
Enis Soztutar created HBASE-6815: Summary: [WINDOWS] Provide hbase scripts in order to start HBASE on Windows in a single user mode Key: HBASE-6815 URL: https://issues.apache.org/jira/browse/HBASE-6815 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Provide .cmd scripts in order to start HBASE on Windows in a single user mode -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6814) [WINDOWS] HBase on Windows
Enis Soztutar created HBASE-6814: Summary: [WINDOWS] HBase on Windows Key: HBASE-6814 URL: https://issues.apache.org/jira/browse/HBASE-6814 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0, 0.94.3 Reporter: Enis Soztutar Assignee: Enis Soztutar This is an umbrella jira to support windows as a first class citizen. The short term goals are: - Get unit tests passing - Scripts converted to windows - Runtime does not depend on cgywin (build can still depend on cygwin for now) - Hbase on windows will consume hadoop branch-1-win artifacts. - Get nightly jenkins build on windows -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6592) [shell] Add means of custom formatting output by column
[ https://issues.apache.org/jira/browse/HBASE-6592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458266#comment-13458266 ] Hudson commented on HBASE-6592: --- Integrated in HBase-TRUNK #3350 (See [https://builds.apache.org/job/HBase-TRUNK/3350/]) HBASE-6592 [shell] Add means of custom formatting output by column (Revision 1387369) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/ruby/hbase/table.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/get.rb * /hbase/trunk/hbase-server/src/main/ruby/shell/commands/scan.rb * /hbase/trunk/hbase-server/src/test/ruby/hbase/table_test.rb > [shell] Add means of custom formatting output by column > --- > > Key: HBASE-6592 > URL: https://issues.apache.org/jira/browse/HBASE-6592 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: stack >Assignee: Jie Huang >Priority: Minor > Labels: noob > Fix For: 0.96.0 > > Attachments: 6592v3.txt, hbase-6592.patch, hbase-6592-v2.patch, > hbase-6952-v1.patch > > > See Jacques suggestion toward end of this thread for how we should allow > adding a custom formatter per column to use outputting column content in > shell: > http://search-hadoop.com/m/2WxUB1fuxL11/Printing+integers+in+the+Hbase+shell&subj=Printing+integers+in+the+Hbase+shell -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6779) Fix issues analysis.apache.org raises about StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-6779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458267#comment-13458267 ] Hudson commented on HBASE-6779: --- Integrated in HBase-TRUNK #3350 (See [https://builds.apache.org/job/HBase-TRUNK/3350/]) HBASE-6779 Fix issues analysis.apache.org raises about StochasticLoadBalancer (Revision 1387372) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java > Fix issues analysis.apache.org raises about StochasticLoadBalancer > -- > > Key: HBASE-6779 > URL: https://issues.apache.org/jira/browse/HBASE-6779 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.96.0 > > Attachments: HBASE-6779-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6649) [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1]
[ https://issues.apache.org/jira/browse/HBASE-6649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458251#comment-13458251 ] Devaraj Das commented on HBASE-6649: Has there been any change in your cluster environment (hadoop version, etc. using different version of dfs client causing the issue to surface)? [Not sure which hadoop version you are on, but there is no chance you are hitting HDFS-1108, right?] > [0.92 UNIT TESTS] TestReplication.queueFailover occasionally fails [Part-1] > --- > > Key: HBASE-6649 > URL: https://issues.apache.org/jira/browse/HBASE-6649 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0, 0.92.3, 0.94.2 > > Attachments: 6649-0.92.patch, 6649-1.patch, 6649-2.txt, > 6649-trunk.patch, 6649-trunk.patch, 6649.txt, HBase-0.92 #495 test - > queueFailover [Jenkins].html, HBase-0.92 #502 test - queueFailover > [Jenkins].html > > > Have seen it twice in the recent past: http://bit.ly/MPCykB & > http://bit.ly/O79Dq7 .. > Looking briefly at the logs hints at a pattern - in both the failed test > instances, there was an RS crash while the test was running. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6813) Optimise the time spent holding the updateLock under log roll
Amitanand Aiyer created HBASE-6813: -- Summary: Optimise the time spent holding the updateLock under log roll Key: HBASE-6813 URL: https://issues.apache.org/jira/browse/HBASE-6813 Project: HBase Issue Type: Improvement Reporter: Amitanand Aiyer Priority: Minor Log roll entails syncing the old log, closing it and creating a new log file. We currently do all the 3 steps while holding the updateLock. This causes latency spikes for puts during this time. We only need to sync the old log under the lock. Creating the new file, can be done before grabbing the lock. Closing the old file can be done after we release the lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6789) Convert test CoprocessorProtocol implementations to protocol buffer services
[ https://issues.apache.org/jira/browse/HBASE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458241#comment-13458241 ] Gary Helmling commented on HBASE-6789: -- This would be a blocker if we want to completely drop CoprocessorProtocol implementation support in 0.96 as opposed to deprecate. Though I do feel that we should convert all of the implementations we ship (they can be dual CoprocessorProtocol/CoprocessorService implementations currently), it's not strictly required if support is still there. > Convert test CoprocessorProtocol implementations to protocol buffer services > > > Key: HBASE-6789 > URL: https://issues.apache.org/jira/browse/HBASE-6789 > Project: HBase > Issue Type: Sub-task > Components: coprocessors >Reporter: Gary Helmling > Fix For: 0.96.0 > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > Several CoprocessorProtocol implementations are defined for tests: > * ColumnAggregationProtocol > * GenericProtocol > * TestServerCustomProtocol.PingProtocol > These should either be converted to PB services or removed if they duplicate > other tests/are no longer necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6591) checkAndPut executed/not metrics
[ https://issues.apache.org/jira/browse/HBASE-6591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458240#comment-13458240 ] Gregory Chanan commented on HBASE-6591: --- Looks like failing hadoop-2.0 build. Doubt it's due to this patch. Will investigate. > checkAndPut executed/not metrics > > > Key: HBASE-6591 > URL: https://issues.apache.org/jira/browse/HBASE-6591 > Project: HBase > Issue Type: Task > Components: metrics, regionserver >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-6591.patch, HBASE-6591-v2.patch, > HBASE-6591-v2.patch, HBASE-6591-v3.patch > > > checkAndPut/checkAndDelete return true if the new put was executed, false > otherwise. > So clients can figure out this metric for themselves, but it would be useful > to get a look at what is happening on the cluster as a whole, across all > clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458236#comment-13458236 ] Gary Helmling commented on HBASE-6788: -- Yes, this one definitely. The other conversion sub-tasks would be blockers as well if we're actually going to drop the CoprocessorProtocol implementations from 0.96 instead of deprecate. > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: coprocessors >Reporter: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6788) Convert AuthenticationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6788: - Priority: Blocker (was: Major) > Convert AuthenticationProtocol to protocol buffer service > - > > Key: HBASE-6788 > URL: https://issues.apache.org/jira/browse/HBASE-6788 > Project: HBase > Issue Type: Sub-task > Components: coprocessors >Reporter: Gary Helmling >Priority: Blocker > Fix For: 0.96.0 > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. > AccessControllerProtocol was converted as part of HBASE-5448, but the > authentication token provider still needs to be changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6408) Naming and documenting of the hadoop-metrics2.properties file
[ https://issues.apache.org/jira/browse/HBASE-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458234#comment-13458234 ] Elliott Clark commented on HBASE-6408: -- Nope that *.file*.sink part just says that all file sinks will use a period of 10 seconds. Since there is no filename and no explicit name for the sink nothing will be created. > Naming and documenting of the hadoop-metrics2.properties file > - > > Key: HBASE-6408 > URL: https://issues.apache.org/jira/browse/HBASE-6408 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Attachments: HBASE-6408-0.patch > > > hadoop-metrics2.properties is currently where metrics2 loads it's sinks. > This file could be better named, hadoop-hbase-metrics2.properties > In addition it needs examples like the current hadoop-metrics.properties has. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6408) Naming and documenting of the hadoop-metrics2.properties file
[ https://issues.apache.org/jira/browse/HBASE-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458232#comment-13458232 ] stack commented on HBASE-6408: -- This patch has FileSink uncommented. Is that right? Does that mean that we will be writing metrics to a file on start up when this patch is applied? > Naming and documenting of the hadoop-metrics2.properties file > - > > Key: HBASE-6408 > URL: https://issues.apache.org/jira/browse/HBASE-6408 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Attachments: HBASE-6408-0.patch > > > hadoop-metrics2.properties is currently where metrics2 loads it's sinks. > This file could be better named, hadoop-hbase-metrics2.properties > In addition it needs examples like the current hadoop-metrics.properties has. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
