[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478736#comment-13478736
]
Hadoop QA commented on HBASE-5498:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549633/HBASE-5498_trunk_3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
83 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//console
This message is automatically generated.
> Secure Bulk Load
>
>
> Key: HBASE-5498
> URL: https://issues.apache.org/jira/browse/HBASE-5498
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Francis Liu
>Assignee: Francis Liu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch,
> HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch,
> HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch,
> HBASE-5498_trunk.patch
>
>
> Design doc:
> https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
> Short summary:
> Security as it stands does not cover the bulkLoadHFiles() feature. Users
> calling this method will bypass ACLs. Also loading is made more cumbersome in
> a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
> from user's directory to the hbase directory, which would require certain
> write access privileges set.
> Our solution is to create a coprocessor which makes use of AuthManager to
> verify if a user has write access to the table. If so, launches a MR job as
> the hbase user to do the importing (ie rewrite from text to hfiles). One
> tricky part this job will have to do is impersonate the calling user when
> reading the input files. We can do this by expecting the user to pass an hdfs
> delegation token as part of the secureBulkLoad() coprocessor call and extend
> an inputformat to make use of that token. The output is written to a
> temporary directory accessible only by hbase and then bulkloadHFiles() is
> called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478733#comment-13478733 ] Hudson commented on HBASE-7002: --- Integrated in HBase-TRUNK #3455 (See [https://builds.apache.org/job/HBase-TRUNK/3455/]) HBASE-7002 Fix all 4 findbug performance warnings (Liang Xie) (Revision 1399514) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java > Fix all 4 findbug performance warnings > -- > > Key: HBASE-7002 > URL: https://issues.apache.org/jira/browse/HBASE-7002 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: liang xie >Assignee: liang xie >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7002.patch > > > Fix the perf warning from this report : > https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478732#comment-13478732 ] Hudson commented on HBASE-6758: --- Integrated in HBase-TRUNK #3455 (See [https://builds.apache.org/job/HBase-TRUNK/3455/]) HBASE-6758 [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file (Devaraj Das via JD) (Revision 1399517) Result = FAILURE jdcryans : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java > [replication] The replication-executor should make sure the file that it is > replicating is closed before declaring success on that file > --- > > Key: HBASE-6758 > URL: https://issues.apache.org/jira/browse/HBASE-6758 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, > 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, > 6758-trunk-4.patch, > TEST-org.apache.hadoop.hbase.replication.TestReplication.xml > > > I have seen cases where the replication-executor would lose data to replicate > since the file hasn't been closed yet. Upon closing, the new data becomes > visible. Before that happens the ZK node shouldn't be deleted in > ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made > in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6786: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, DD. > Convert MultiRowMutationProtocol to protocol buffer service > --- > > Key: HBASE-6786 > URL: https://issues.apache.org/jira/browse/HBASE-6786 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6786-1.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Assignee: liang xie Status: Patch Available (was: Open) > Set scanner caching to a better default > --- > > Key: HBASE-7008 > URL: https://issues.apache.org/jira/browse/HBASE-7008 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: liang xie >Assignee: liang xie > Attachments: HBASE-7008.patch > > > per > http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+ > let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Description: per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default > Set scanner caching to a better default > --- > > Key: HBASE-7008 > URL: https://issues.apache.org/jira/browse/HBASE-7008 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: liang xie > Attachments: HBASE-7008.patch > > > per > http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+ > let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Attachment: HBASE-7008.patch > Set scanner caching to a better default > --- > > Key: HBASE-7008 > URL: https://issues.apache.org/jira/browse/HBASE-7008 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: liang xie > Attachments: HBASE-7008.patch > > > per > http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+ > let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478714#comment-13478714 ] Devaraj Das commented on HBASE-6758: This should mostly be applicable on 0.94 straightaway.. > [replication] The replication-executor should make sure the file that it is > replicating is closed before declaring success on that file > --- > > Key: HBASE-6758 > URL: https://issues.apache.org/jira/browse/HBASE-6758 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, > 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, > 6758-trunk-4.patch, > TEST-org.apache.hadoop.hbase.replication.TestReplication.xml > > > I have seen cases where the replication-executor would lose data to replicate > since the file hasn't been closed yet. Upon closing, the new data becomes > visible. Before that happens the ZK node shouldn't be deleted in > ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made > in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7008) Set scanner caching to a better default
liang xie created HBASE-7008: Summary: Set scanner caching to a better default Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478695#comment-13478695 ] Lars Hofhansl commented on HBASE-6758: -- 0.94? Looks like a good fix to backport. > [replication] The replication-executor should make sure the file that it is > replicating is closed before declaring success on that file > --- > > Key: HBASE-6758 > URL: https://issues.apache.org/jira/browse/HBASE-6758 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, > 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, > 6758-trunk-4.patch, > TEST-org.apache.hadoop.hbase.replication.TestReplication.xml > > > I have seen cases where the replication-executor would lose data to replicate > since the file hasn't been closed yet. Upon closing, the new data becomes > visible. Before that happens the ZK node shouldn't be deleted in > ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made > in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478694#comment-13478694
]
Hadoop QA commented on HBASE-5498:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549633/HBASE-5498_trunk_3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
83 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3073//console
This message is automatically generated.
> Secure Bulk Load
>
>
> Key: HBASE-5498
> URL: https://issues.apache.org/jira/browse/HBASE-5498
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Francis Liu
>Assignee: Francis Liu
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch,
> HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch,
> HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch,
> HBASE-5498_trunk.patch
>
>
> Design doc:
> https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
> Short summary:
> Security as it stands does not cover the bulkLoadHFiles() feature. Users
> calling this method will bypass ACLs. Also loading is made more cumbersome in
> a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
> from user's directory to the hbase directory, which would require certain
> write access privileges set.
> Our solution is to create a coprocessor which makes use of AuthManager to
> verify if a user has write access to the table. If so, launches a MR job as
> the hbase user to do the importing (ie rewrite from text to hfiles). One
> tricky part this job will have to do is impersonate the calling user when
> reading the input files. We can do this by expecting the user to pass an hdfs
> delegation token as part of the secureBulkLoad() coprocessor call and extend
> an inputformat to make use of that token. The output is written to a
> temporary directory accessible only by hbase and then bulkloadHFiles() is
> called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478690#comment-13478690 ] Lars Hofhansl commented on HBASE-6032: -- Whoops. Forgot to "svn add" the two new files. Done through an addendum. > Port HFileBlockIndex improvement from HBASE-5987 > > > Key: HBASE-6032 > URL: https://issues.apache.org/jira/browse/HBASE-6032 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.94.3, 0.96.0 > > Attachments: 6032.094.txt, 6032-ports-5987.txt, > 6032-ports-5987-v2.txt, 6032v3.txt > > > Excerpt from HBASE-5987: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478689#comment-13478689 ] Gary Helmling commented on HBASE-6786: -- +1 Patch looks good to me. Thanks, Deveraj. > Convert MultiRowMutationProtocol to protocol buffer service > --- > > Key: HBASE-6786 > URL: https://issues.apache.org/jira/browse/HBASE-6786 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6786-1.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478679#comment-13478679 ] Hudson commented on HBASE-5987: --- Integrated in HBase-0.94 #539 (See [https://builds.apache.org/job/HBase-0.94/539/]) HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java > HFileBlockIndex improvement > --- > > Key: HBASE-5987 > URL: https://issues.apache.org/jira/browse/HBASE-5987 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > Fix For: 0.96.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--D3237.1.patch, > ASF.LICENSE.NOT.GRANTED--D3237.2.patch, > ASF.LICENSE.NOT.GRANTED--D3237.3.patch, > ASF.LICENSE.NOT.GRANTED--D3237.4.patch, > ASF.LICENSE.NOT.GRANTED--D3237.5.patch, > ASF.LICENSE.NOT.GRANTED--D3237.6.patch, > ASF.LICENSE.NOT.GRANTED--D3237.7.patch, > ASF.LICENSE.NOT.GRANTED--D3237.8.patch, > screen_shot_of_sequential_scan_profiling.png > > > Recently we find out a performance problem that it is quite slow when > multiple requests are reading the same block of data or index. > From the profiling, one of the causes is the IdLock contention which has been > addressed in HBASE-5898. > Another issue is that the HFileScanner will keep asking the HFileBlockIndex > about the data block location for each target key value during the scan > process(reSeekTo), even though the target key value has already been in the > current data block. This issue will cause certain index block very HOT, > especially when it is a sequential scan. > To solve this issue, we propose the following solutions: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > Secondary, we propose to push this idea a little further that the > HFileBlockIndex shall index on the last key value of each data block instead > of indexing on the start key value. The motivation is to solve the HBASE-4443 > issue (avoid seeking to "previous" block when key you are interested in is > the first one of a block) as well as +the defects mentioned above+. > For example, if the target key value is "smaller" than the start key value of > the data block N. There is no way for sure the target key value is in the > data block N or N-1. So it has to seek from data block N-1. However, if the > block index is based on the last key value for each data block and the target > key value is beween the last key value of data block N-1 and data block N, > then the target key value is supposed be data block N for sure. > As long as HBase only supports the forward scan, the last key value makes > more sense to be indexed on than the start key value. > Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478678#comment-13478678 ] Hudson commented on HBASE-6032: --- Integrated in HBase-0.94 #539 (See [https://builds.apache.org/job/HBase-0.94/539/]) HBASE-6032 Port HFileBlockIndex improvement from HBASE-5987 (Liyin, Ted, Stack) (Revision 1399513) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java > Port HFileBlockIndex improvement from HBASE-5987 > > > Key: HBASE-6032 > URL: https://issues.apache.org/jira/browse/HBASE-6032 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.94.3, 0.96.0 > > Attachments: 6032.094.txt, 6032-ports-5987.txt, > 6032-ports-5987-v2.txt, 6032v3.txt > > > Excerpt from HBASE-5987: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478677#comment-13478677 ] Devaraj Das commented on HBASE-6758: Thanks, [~jdcryans], for the reviews. Party time :-) > [replication] The replication-executor should make sure the file that it is > replicating is closed before declaring success on that file > --- > > Key: HBASE-6758 > URL: https://issues.apache.org/jira/browse/HBASE-6758 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, > 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, > 6758-trunk-4.patch, > TEST-org.apache.hadoop.hbase.replication.TestReplication.xml > > > I have seen cases where the replication-executor would lose data to replicate > since the file hasn't been closed yet. Upon closing, the new data becomes > visible. Before that happens the ZK node shouldn't be deleted in > ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made > in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6987) Port HBASE-6920 to trunk (?)
[
https://issues.apache.org/jira/browse/HBASE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478675#comment-13478675
]
Hadoop QA commented on HBASE-6987:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549624/HBASE-6987.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3072//console
This message is automatically generated.
> Port HBASE-6920 to trunk (?)
>
>
> Key: HBASE-6987
> URL: https://issues.apache.org/jira/browse/HBASE-6987
> Project: HBase
> Issue Type: Bug
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-6987.patch
>
>
> Need to investigate whether we need to port HBASE-6920 to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-6758: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for your patience and patch Devaraj. > [replication] The replication-executor should make sure the file that it is > replicating is closed before declaring success on that file > --- > > Key: HBASE-6758 > URL: https://issues.apache.org/jira/browse/HBASE-6758 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, > 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, > 6758-trunk-4.patch, > TEST-org.apache.hadoop.hbase.replication.TestReplication.xml > > > I have seen cases where the replication-executor would lose data to replicate > since the file hasn't been closed yet. Upon closing, the new data becomes > visible. Before that happens the ZK node shouldn't be deleted in > ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made > in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7002: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Fix all 4 findbug performance warnings > -- > > Key: HBASE-7002 > URL: https://issues.apache.org/jira/browse/HBASE-7002 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: liang xie >Assignee: liang xie >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7002.patch > > > Fix the perf warning from this report : > https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478653#comment-13478653 ] Ted Yu commented on HBASE-7002: --- Integrated to trunk. Thanks for the patch, Liang. > Fix all 4 findbug performance warnings > -- > > Key: HBASE-7002 > URL: https://issues.apache.org/jira/browse/HBASE-7002 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: liang xie >Assignee: liang xie >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7002.patch > > > Fix the perf warning from this report : > https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7002: -- Fix Version/s: 0.96.0 Hadoop Flags: Reviewed > Fix all 4 findbug performance warnings > -- > > Key: HBASE-7002 > URL: https://issues.apache.org/jira/browse/HBASE-7002 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: liang xie >Assignee: liang xie >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7002.patch > > > Fix the perf warning from this report : > https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6997) HRegion.mutateRows() should check for blocking memstore size and if the region is closing.
[ https://issues.apache.org/jira/browse/HBASE-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6997: - Fix Version/s: (was: 0.94.3) (was: 0.96.0) > HRegion.mutateRows() should check for blocking memstore size and if the > region is closing. > -- > > Key: HBASE-6997 > URL: https://issues.apache.org/jira/browse/HBASE-6997 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > Currently HRegion.mutateRows() does not call startRegionOperation() and > checkResources(). Ideally before doing mutateRows() the memstore size and > 'closing' flag should be checked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6986) Reenable TestClientTimeouts for security build
[ https://issues.apache.org/jira/browse/HBASE-6986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6986: - Fix Version/s: (was: 0.94.3) > Reenable TestClientTimeouts for security build > -- > > Key: HBASE-6986 > URL: https://issues.apache.org/jira/browse/HBASE-6986 > Project: HBase > Issue Type: Sub-task >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > > TestClientTimeouts was disabled to get 0.94.2 out the door because it didn't > work in security build. Investigate and reenable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7000) Fix the "INT_VACUOUS_COMPARISON" WARNING in KeyValue class
[
https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478647#comment-13478647
]
Hadoop QA commented on HBASE-7000:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549626/HBASE-7000-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3071//console
This message is automatically generated.
> Fix the "INT_VACUOUS_COMPARISON" WARNING in KeyValue class
> --
>
> Key: HBASE-7000
> URL: https://issues.apache.org/jira/browse/HBASE-7000
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: liang xie
>Assignee: liang xie
>Priority: Minor
> Attachments: HBASE-7000.patch, HBASE-7000-v2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478646#comment-13478646 ] Ted Yu commented on HBASE-5498: --- To convert SecureBulkLoadProtocol to PB, you can use HBASE-6785 'Convert AggregateProtocol to protobuf defined coprocessor service' as an example. prepareBulkLoad() and cleanupBulkLoad() are relatively straightforward. For bulkLoadHFiles(), please refer to message BulkLoadHFileRequest in Client.proto Thanks > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, > HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, > HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, > HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6032: - Fix Version/s: 0.94.3 Committed to 0.94 as well. > Port HFileBlockIndex improvement from HBASE-5987 > > > Key: HBASE-6032 > URL: https://issues.apache.org/jira/browse/HBASE-6032 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.94.3, 0.96.0 > > Attachments: 6032.094.txt, 6032-ports-5987.txt, > 6032-ports-5987-v2.txt, 6032v3.txt > > > Excerpt from HBASE-5987: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478643#comment-13478643 ] binlijin commented on HBASE-5987: - @Lars Hofhansl: Thank you very much. > HFileBlockIndex improvement > --- > > Key: HBASE-5987 > URL: https://issues.apache.org/jira/browse/HBASE-5987 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > Fix For: 0.96.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--D3237.1.patch, > ASF.LICENSE.NOT.GRANTED--D3237.2.patch, > ASF.LICENSE.NOT.GRANTED--D3237.3.patch, > ASF.LICENSE.NOT.GRANTED--D3237.4.patch, > ASF.LICENSE.NOT.GRANTED--D3237.5.patch, > ASF.LICENSE.NOT.GRANTED--D3237.6.patch, > ASF.LICENSE.NOT.GRANTED--D3237.7.patch, > ASF.LICENSE.NOT.GRANTED--D3237.8.patch, > screen_shot_of_sequential_scan_profiling.png > > > Recently we find out a performance problem that it is quite slow when > multiple requests are reading the same block of data or index. > From the profiling, one of the causes is the IdLock contention which has been > addressed in HBASE-5898. > Another issue is that the HFileScanner will keep asking the HFileBlockIndex > about the data block location for each target key value during the scan > process(reSeekTo), even though the target key value has already been in the > current data block. This issue will cause certain index block very HOT, > especially when it is a sequential scan. > To solve this issue, we propose the following solutions: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > Secondary, we propose to push this idea a little further that the > HFileBlockIndex shall index on the last key value of each data block instead > of indexing on the start key value. The motivation is to solve the HBASE-4443 > issue (avoid seeking to "previous" block when key you are interested in is > the first one of a block) as well as +the defects mentioned above+. > For example, if the target key value is "smaller" than the start key value of > the data block N. There is no way for sure the target key value is in the > data block N or N-1. So it has to seek from data block N-1. However, if the > block index is based on the last key value for each data block and the target > key value is beween the last key value of data block N-1 and data block N, > then the target key value is supposed be data block N for sure. > As long as HBase only supports the forward scan, the last key value makes > more sense to be indexed on than the start key value. > Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5987) HFileBlockIndex improvement
[ https://issues.apache.org/jira/browse/HBASE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478636#comment-13478636 ] Lars Hofhansl commented on HBASE-5987: -- @binlijin: We're doing the porting in HBASE-6032. > HFileBlockIndex improvement > --- > > Key: HBASE-5987 > URL: https://issues.apache.org/jira/browse/HBASE-5987 > Project: HBase > Issue Type: Improvement >Reporter: Liyin Tang >Assignee: Liyin Tang > Fix For: 0.96.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--D3237.1.patch, > ASF.LICENSE.NOT.GRANTED--D3237.2.patch, > ASF.LICENSE.NOT.GRANTED--D3237.3.patch, > ASF.LICENSE.NOT.GRANTED--D3237.4.patch, > ASF.LICENSE.NOT.GRANTED--D3237.5.patch, > ASF.LICENSE.NOT.GRANTED--D3237.6.patch, > ASF.LICENSE.NOT.GRANTED--D3237.7.patch, > ASF.LICENSE.NOT.GRANTED--D3237.8.patch, > screen_shot_of_sequential_scan_profiling.png > > > Recently we find out a performance problem that it is quite slow when > multiple requests are reading the same block of data or index. > From the profiling, one of the causes is the IdLock contention which has been > addressed in HBASE-5898. > Another issue is that the HFileScanner will keep asking the HFileBlockIndex > about the data block location for each target key value during the scan > process(reSeekTo), even though the target key value has already been in the > current data block. This issue will cause certain index block very HOT, > especially when it is a sequential scan. > To solve this issue, we propose the following solutions: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > Secondary, we propose to push this idea a little further that the > HFileBlockIndex shall index on the last key value of each data block instead > of indexing on the start key value. The motivation is to solve the HBASE-4443 > issue (avoid seeking to "previous" block when key you are interested in is > the first one of a block) as well as +the defects mentioned above+. > For example, if the target key value is "smaller" than the start key value of > the data block N. There is no way for sure the target key value is in the > data block N or N-1. So it has to seek from data block N-1. However, if the > block index is based on the last key value for each data block and the target > key value is beween the last key value of data block N-1 and data block N, > then the target key value is supposed be data block N for sure. > As long as HBase only supports the forward scan, the last key value makes > more sense to be indexed on than the start key value. > Thanks Kannan and Mikhail for the insightful discussions and suggestions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478630#comment-13478630 ] Francis Liu commented on HBASE-5498: Hi Guys, can we get a resolution on wether I have to convert the trunk patch to the new implementation? > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, > HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, > HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, > HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_trunk_3.patch fixed a filename typo. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, > HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, > HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch, > HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6980) Parallel Flushing Of Memstores
[ https://issues.apache.org/jira/browse/HBASE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478622#comment-13478622 ] Kannan Muthukkaruppan commented on HBASE-6980: -- Todd: If RS zk expires, and master initiates recovery/log splitting, then the first step is to rename the log directory from .logs/rs to .logs/rs-splitting. And then the lease recovery is done on the individual files within the directory. Because of the directory name, any attempt by the old RS to delete any old log files (in the old path) should fail. Therefore, still not seeing the value of writing the flush marker. > Parallel Flushing Of Memstores > -- > > Key: HBASE-6980 > URL: https://issues.apache.org/jira/browse/HBASE-6980 > Project: HBase > Issue Type: New Feature >Reporter: Kannan Muthukkaruppan >Assignee: Kannan Muthukkaruppan > > For write dominated workloads, single threaded memstore flushing is an > unnecessary bottleneck. With a single flusher thread, we are basically not > setup to take advantage of the aggregate throughput that multi-disk nodes > provide. > * For puts with WAL enabled, the bottleneck is more likely the "single" WAL > per region server. So this particular fix may not buy as much unless we > unlock that bottleneck with multiple commit logs per region server. (Topic > for a separate JIRA-- HBASE-6981). > * But for puts with WAL disabled (e.g., when using HBASE-5783 style fast bulk > imports), we should be able to support much better ingest rates with parallel > flushing of memstores. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6942) Endpoint implementation for bulk delete rows
[
https://issues.apache.org/jira/browse/HBASE-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478621#comment-13478621
]
Anoop Sam John commented on HBASE-6942:
---
Yes 0.94 code base
Pls see the javadoc in HRegionInterface.java
{code}
/**
* Put an array of puts into the specified region
*
* @param regionName region name
* @param puts List of puts to execute
* @return The number of processed put's. Returns -1 if all Puts
* processed successfully.
* @throws IOException e
*/
public int put(final byte[] regionName, final List puts)
throws IOException;
{code}
*return The number of processed put's*
> Endpoint implementation for bulk delete rows
>
>
> Key: HBASE-6942
> URL: https://issues.apache.org/jira/browse/HBASE-6942
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, Performance
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6942_DeleteTemplate.patch, HBASE-6942.patch,
> HBASE-6942_V2.patch, HBASE-6942_V3.patch, HBASE-6942_V4.patch,
> HBASE-6942_V5.patch, HBASE-6942_V6.patch
>
>
> We can provide an end point implementation for doing a bulk deletion of
> rows(based on a scan) at the server side. This can reduce the time taken for
> such an operation as right now it need to do a scan to client and issue
> delete(s) using rowkeys.
> Query like delete from table1 where...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[ https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478618#comment-13478618 ] liang xie commented on HBASE-7005: -- +1 > Upgrade Thrift lib to 0.9.0 > --- > > Key: HBASE-7005 > URL: https://issues.apache.org/jira/browse/HBASE-7005 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jake Farrell >Priority: Minor > Attachments: Hbase-7005.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-6411: - Fix Version/s: 0.96.0 > Move Master Metrics to metrics 2 > > > Key: HBASE-6411 > URL: https://issues.apache.org/jira/browse/HBASE-6411 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Alex Baranau > Fix For: 0.96.0 > > Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, > HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411-4_2.patch, > HBASE-6411-4.patch, HBASE-6411-5.patch, HBASE-6411_concept.patch > > > Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6651) Thread safety of HTablePool is doubtful
[ https://issues.apache.org/jira/browse/HBASE-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478592#comment-13478592 ] Hiroshi Ikeda commented on HBASE-6651: -- bq. Can you explain why calling clear() followed by returnObject() would result in return value of false ? There is space in SharedMap at this moment, right ? SharedMap holds a collection of registered objects independent of whether the objects are pooled or borrowed, and SharedMap.invalidateObject() and SharedMap.clear() remove the specified object(s) from the collection. SharedMap.returnObject() always does nothing and returns false if the given object is not found in the collection. Filling a space of the registered objects is done by SharedMap.registerObject(), not SharedMap.returnObject(). > Thread safety of HTablePool is doubtful > --- > > Key: HBASE-6651 > URL: https://issues.apache.org/jira/browse/HBASE-6651 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.94.1 >Reporter: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-6651.patch, HBASE-6651-V2.patch, sample.zip, > sample.zip, sharedmap_for_hbaseclient.zip > > > There are some operations in HTablePool to access to PoolMap in multiple > times without any explict synchronization. > For example HTablePool.closeTablePool() calles PoolMap.values(), and calles > PoolMap.remove(). If other threads add new instances to the pool in the > middle of the calls, the new added instances might be dropped. > (HTablePool.closeTablePool() also has another problem that calling it by > multple threads causes accessing HTable by multiple threads.) > Moreover, PoolMap is not thread safe for the same reason. > For example PoolMap.put() calles ConcurrentMap.get() and calles > ConcurrentMap.put(). If other threads add a new instance to the concurent map > in the middle of the calls, the new instance might be dropped. > And also implementations of Pool have the same problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6611) Forcing region state offline cause double assignment
[ https://issues.apache.org/jira/browse/HBASE-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478567#comment-13478567 ] Jimmy Xiang commented on HBASE-6611: @Stack, @Ram, are we ok with the latest patch? > Forcing region state offline cause double assignment > > > Key: HBASE-6611 > URL: https://issues.apache.org/jira/browse/HBASE-6611 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: trunk-6611_v2.patch > > > In assigning a region, assignment manager forces the region state offline if > it is not. This could cause double assignment, for example, if the region is > already assigned and in the Open state, you should not just change it's state > to Offline, and assign it again. > I think this could be the root cause for all double assignments IF the region > state is reliable. > After this loophole is closed, TestHBaseFsck should come up a different way > to create some assignment inconsistencies, for example, calling region server > to open a region directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478566#comment-13478566 ] Roman Shaposhnik commented on HBASE-6929: - bq. If between hadoop1 and hadoop2, then I don't think we can resolve this easily It is not super trivial, but it isn't insurmountable either. Besides, we can borrow generously from some of the projects that did some of it already. In short -- it definitely can be done, but it requires more than just yours truly ;-) > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7000) Fix the "INT_VACUOUS_COMPARISON" WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7000: - Attachment: HBASE-7000-v2.patch Attached v2 patch addressed Ted's comments > Fix the "INT_VACUOUS_COMPARISON" WARNING in KeyValue class > -- > > Key: HBASE-7000 > URL: https://issues.apache.org/jira/browse/HBASE-7000 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: liang xie >Assignee: liang xie >Priority: Minor > Attachments: HBASE-7000.patch, HBASE-7000-v2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6987) Port HBASE-6920 to trunk (?)
[ https://issues.apache.org/jira/browse/HBASE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6987: -- Status: Patch Available (was: Open) > Port HBASE-6920 to trunk (?) > > > Key: HBASE-6987 > URL: https://issues.apache.org/jira/browse/HBASE-6987 > Project: HBase > Issue Type: Bug >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-6987.patch > > > Need to investigate whether we need to port HBASE-6920 to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6987) Port HBASE-6920 to trunk (?)
[ https://issues.apache.org/jira/browse/HBASE-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6987: -- Attachment: HBASE-6987.patch * Attached HBASE-6987.patch * Code in trunk looks good (thanks again to nkeywal). This patch makes a couple of changes: - Simplifies some HBaseAdmin logic wrt checking if the master is available - Adds a test for HBASE-6920 (it passes, since the code-under-test is good :)). It is non-invasive to the RpcEngine layer, unlike the original test in HBASE-6920. The test isn't perfect; it only throws ServiceExceptions because that is all that ProtobufRpcEngine throws. If that changes or another RpcEngine is used, the RandomTimeoutRpcEngine may not properly mimic the exception types coming from the RpcEngine. Putting hooks into any deeper layer (e.g. HBaseClient) would get very tricky and invasive. > Port HBASE-6920 to trunk (?) > > > Key: HBASE-6987 > URL: https://issues.apache.org/jira/browse/HBASE-6987 > Project: HBase > Issue Type: Bug >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-6987.patch > > > Need to investigate whether we need to port HBASE-6920 to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6371) [89-fb] Tier based compaction
[ https://issues.apache.org/jira/browse/HBASE-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478543#comment-13478543 ] Lars Hofhansl commented on HBASE-6371: -- Specifically a scenario I'd be interested in, is to keep a days (or two) worth of changes in a live HBase cluster. In extreme cases this might be lead to 1000's of versions, and scan performance of the latest version suffers significantly, *especially* after a major compaction which will cause all version of KVs to be jumbled together in the same file. > [89-fb] Tier based compaction > - > > Key: HBASE-6371 > URL: https://issues.apache.org/jira/browse/HBASE-6371 > Project: HBase > Issue Type: Improvement >Reporter: Akashnil >Assignee: Liyin Tang > Labels: noob > > Currently, the compaction selection is not very flexible and is not sensitive > to the hotness of the data. Very old data is likely to be accessed less, and > very recent data is likely to be in the block cache. Both of these > considerations make it inefficient to compact these files as aggressively as > other files. In some use-cases, the access-pattern is particularly obvious > even though there is no way to control the compaction algorithm in those > cases. > In the new compaction selection algorithm, we plan to divide the candidate > files into different levels according to oldness of the data that is present > in those files. For each level, parameters like compaction ratio, minimum > number of store-files in each compaction may be different. Number of levels, > time-ranges, and parameters for each level will be configurable online on a > per-column family basis. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6577) RegionScannerImpl.nextRow() should seek to next row
[ https://issues.apache.org/jira/browse/HBASE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478539#comment-13478539 ] Lars Hofhansl commented on HBASE-6577: -- @J-D: with the initial version of this patch you encountered this problem on your production cluster: https://issues.apache.org/jira/browse/HBASE-6577?focusedCommentId=13440843&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13440843 And https://issues.apache.org/jira/browse/HBASE-6577?focusedCommentId=13440924&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13440924 Just wanna make sure this does not happen again with this patch. > RegionScannerImpl.nextRow() should seek to next row > --- > > Key: HBASE-6577 > URL: https://issues.apache.org/jira/browse/HBASE-6577 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0 > > Attachments: 6577-0.94.txt, 6577.txt, 6577-v2.txt, 6577-v3.txt > > > RegionScannerImpl.nextRow() is called when a filter filters the entire row. > In that case we should seek to the next row rather then iterating over all > versions of all columns to get there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6371) [89-fb] Tier based compaction
[ https://issues.apache.org/jira/browse/HBASE-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478537#comment-13478537 ] Lars Hofhansl commented on HBASE-6371: -- Is this in the 0.89 branch, and is it finished? Does this only change the compaction selection? A feature I'd be interested in is separating old and new data, such it would not typically be compacted into the same file(s) - even in a major compaction. If there is hot new data and lot of older data that is only occasionally queried, if old and new data ends up in the same set of files, scanning is slow, because scanning the latest version only is hitting many more blocks then necessary and old versions of KVs need to be skipped (and HFiles with only old data could otherwise simply be ignored). An approach LevelDB might be interesting. Here's a description: http://code.google.com/p/leveldb/source/browse/doc/impl.html Is that what we are trying to do with this? > [89-fb] Tier based compaction > - > > Key: HBASE-6371 > URL: https://issues.apache.org/jira/browse/HBASE-6371 > Project: HBase > Issue Type: Improvement >Reporter: Akashnil >Assignee: Liyin Tang > Labels: noob > > Currently, the compaction selection is not very flexible and is not sensitive > to the hotness of the data. Very old data is likely to be accessed less, and > very recent data is likely to be in the block cache. Both of these > considerations make it inefficient to compact these files as aggressively as > other files. In some use-cases, the access-pattern is particularly obvious > even though there is no way to control the compaction algorithm in those > cases. > In the new compaction selection algorithm, we plan to divide the candidate > files into different levels according to oldness of the data that is present > in those files. For each level, parameters like compaction ratio, minimum > number of store-files in each compaction may be different. Number of levels, > time-ranges, and parameters for each level will be configurable online on a > per-column family basis. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_94_3.patch fixed a filename typo. > Secure Bulk Load > > > Key: HBASE-5498 > URL: https://issues.apache.org/jira/browse/HBASE-5498 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch, > HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, > HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk.patch > > > Design doc: > https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load > Short summary: > Security as it stands does not cover the bulkLoadHFiles() feature. Users > calling this method will bypass ACLs. Also loading is made more cumbersome in > a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data > from user's directory to the hbase directory, which would require certain > write access privileges set. > Our solution is to create a coprocessor which makes use of AuthManager to > verify if a user has write access to the table. If so, launches a MR job as > the hbase user to do the importing (ie rewrite from text to hfiles). One > tricky part this job will have to do is impersonate the calling user when > reading the input files. We can do this by expecting the user to pass an hdfs > delegation token as part of the secureBulkLoad() coprocessor call and extend > an inputformat to make use of that token. The output is written to a > temporary directory accessible only by hbase and then bulkloadHFiles() is > called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478518#comment-13478518 ] Enis Soztutar commented on HBASE-6929: -- bq. Another possible solution would be to provide artifact hbase-test that would have classifier either hadoop1 or hadoop2 based on for which version it was compiled We should not do this for h1, since that would break maven compatibility that our previous artifacts (0.94.0, 0.94.1, etc) was released with artifact name hbase, and classifier tests. However, maybe we can do it for hadoop2, not sure how easy it would be to create hbase-tests artifact with classifier hadoop-2. Since maven-deploy hardcodes "tests" classifier, this might not be possible at all. bq. Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then. What versions of hadoop? If between hadoop1 and hadoop2, then I don't think we can resolve this easily without creating a shim layer for pretty much everything. See: http://dbeech.github.com/hadoop-api-evolution.html > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6577) RegionScannerImpl.nextRow() should seek to next row
[ https://issues.apache.org/jira/browse/HBASE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478515#comment-13478515 ] Jean-Daniel Cryans commented on HBASE-6577: --- bq. Jean-Daniel Cryans Would you be able the recreate the issue you saw with the initial version of this patch in production? Is there anything in particular I should look for? > RegionScannerImpl.nextRow() should seek to next row > --- > > Key: HBASE-6577 > URL: https://issues.apache.org/jira/browse/HBASE-6577 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0 > > Attachments: 6577-0.94.txt, 6577.txt, 6577-v2.txt, 6577-v3.txt > > > RegionScannerImpl.nextRow() is called when a filter filters the entire row. > In that case we should seek to the next row rather then iterating over all > versions of all columns to get there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7007) [MTTR] Study assigns to see if we can make them faster still
stack created HBASE-7007: Summary: [MTTR] Study assigns to see if we can make them faster still Key: HBASE-7007 URL: https://issues.apache.org/jira/browse/HBASE-7007 Project: HBase Issue Type: Improvement Reporter: stack Looking at a cluster start, I saw that it took about 25 minutes to assign and open 17k regions. 8 minutes was bulk assigning via zk. 17 minutes was opening the regions. "HBASE-6640 [0.89-fb] Allow multiple regions to be opened simultaneously" in trunk would help but maybe we can do less work up in zk for instance; e.g. if 3.4.5, we can do bulk ops in zk (or make a bulk assign znode that has many regions instead of one) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6640) [0.89-fb] Allow multiple regions to be opened simultaneously
[ https://issues.apache.org/jira/browse/HBASE-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478508#comment-13478508 ] stack commented on HBASE-6640: -- Consider backporting this to 0.94 too. > [0.89-fb] Allow multiple regions to be opened simultaneously > > > Key: HBASE-6640 > URL: https://issues.apache.org/jira/browse/HBASE-6640 > Project: HBase > Issue Type: Improvement >Reporter: Amitanand Aiyer >Assignee: Amitanand Aiyer >Priority: Critical > Fix For: 0.96.0 > > Attachments: > 0001-HBASE-6640-0.89-fb-Allow-multiple-regions-to-be-open.patch > > > Allow regions to be opened in parallel. This should reduce the time it takes > to replay and reopen regions in case of a unclean restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7006) [MTTR] Study distributed log splitting to see how we can make it faster
stack created HBASE-7006: Summary: [MTTR] Study distributed log splitting to see how we can make it faster Key: HBASE-7006 URL: https://issues.apache.org/jira/browse/HBASE-7006 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Fix For: 0.96.0 Just saw interesting issue where a cluster went down hard and 30 nodes had 1700 WALs to replay. Replay took almost an hour. It looks like it could run faster that much of the time is spent zk'ing and nn'ing. Putting in 0.96 so it gets a look at least. Can always punt. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6979) recovered.edits file should not break distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478498#comment-13478498 ] Hudson commented on HBASE-6979: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #225 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/225/]) HBASE-6979 recovered.edits file should not break distributed log splitting (Revision 1399352) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java > recovered.edits file should not break distributed log splitting > --- > > Key: HBASE-6979 > URL: https://issues.apache.org/jira/browse/HBASE-6979 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: trunk-6979.patch > > > Distributed log splitting fails in creating the recovered.edits folder during > upgrade because there is a file called recovered.edits there. > Instead of checking if the patch exists, we need to check if it exists and is > a path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6948) shell create table script cannot handle split key which is expressed in raw bytes
[ https://issues.apache.org/jira/browse/HBASE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478497#comment-13478497 ] Hudson commented on HBASE-6948: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #225 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/225/]) HBASE-6948 shell create table script cannot handle split key which is expressed in raw bytes (Tianying) (Revision 1399429) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb > shell create table script cannot handle split key which is expressed in raw > bytes > - > > Key: HBASE-6948 > URL: https://issues.apache.org/jira/browse/HBASE-6948 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2 >Reporter: Ted Yu >Assignee: Tianying Chang > Fix For: 0.96.0 > > Attachments: HBASE-6948.patch, HBASE-6948-trunk.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478474#comment-13478474 ] Roman Shaposhnik commented on HBASE-6929: - Long time ago I filed HBASE-4850 but haven't had much time to devote to it since then. Is there any chance that somebody can help me working on it? > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478472#comment-13478472 ] Lars Hofhansl commented on HBASE-6974: -- Actually... Should probably name the metric more appropriately. How about "memstoreSizeBlockedSeconds", and "memstoreHighWaterBlockedSeconds"? > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch, HBASE-6974-v4.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[
https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478469#comment-13478469
]
Hadoop QA commented on HBASE-6974:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549585/HBASE-6974-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3070//console
This message is automatically generated.
> Metric for blocked updates
> --
>
> Key: HBASE-6974
> URL: https://issues.apache.org/jira/browse/HBASE-6974
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Michael Drzal
>Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6974.patch, HBASE-6974-v2.patch,
> HBASE-6974-v3.patch, HBASE-6974-v4.patch
>
>
> When the disc subsystem cannot keep up with a sustained high write load, a
> region will eventually block updates to throttle clients.
> (HRegion.checkResources).
> It would be nice to have a metric for this, so that these occurrences can be
> tracked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478468#comment-13478468 ] Lars Hofhansl commented on HBASE-6974: -- This is a low risk change, going to commit soon, unless I hear an objections. > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch, HBASE-6974-v4.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478465#comment-13478465 ] Lars Hofhansl commented on HBASE-6974: -- +1 on v4 (super-duper minor nit: the blocked flag can be pulled into the if statement in the memstore flusher - I'll do that on commit, no need for a new patch). Thanks Drz! > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch, HBASE-6974-v4.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[ https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478450#comment-13478450 ] Jake Farrell commented on HBASE-7005: - Patch used existing test cases as it was only upgrading an existing library. Ran test cases and all passed including the TestZooKeeperTableArchiveClient Running org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.909 sec [INFO] [INFO] Reactor Summary: [INFO] [INFO] HBase . SUCCESS [1.786s] [INFO] HBase - Common SUCCESS [7.448s] [INFO] HBase - Hadoop Compatibility .. SUCCESS [0.497s] [INFO] HBase - Hadoop One Compatibility .. SUCCESS [0.988s] [INFO] HBase - Server SUCCESS [40:31.865s] [INFO] HBase - Hadoop Two Compatibility .. SUCCESS [6.360s] [INFO] HBase - Integration Tests . SUCCESS [1.534s] [INFO] [INFO] BUILD SUCCESS > Upgrade Thrift lib to 0.9.0 > --- > > Key: HBASE-7005 > URL: https://issues.apache.org/jira/browse/HBASE-7005 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jake Farrell >Priority: Minor > Attachments: Hbase-7005.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6948) shell create table script cannot handle split key which is expressed in raw bytes
[ https://issues.apache.org/jira/browse/HBASE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478440#comment-13478440 ] Hudson commented on HBASE-6948: --- Integrated in HBase-TRUNK #3453 (See [https://builds.apache.org/job/HBase-TRUNK/3453/]) HBASE-6948 shell create table script cannot handle split key which is expressed in raw bytes (Tianying) (Revision 1399429) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb > shell create table script cannot handle split key which is expressed in raw > bytes > - > > Key: HBASE-6948 > URL: https://issues.apache.org/jira/browse/HBASE-6948 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2 >Reporter: Ted Yu >Assignee: Tianying Chang > Fix For: 0.96.0 > > Attachments: HBASE-6948.patch, HBASE-6948-trunk.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[
https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478427#comment-13478427
]
Hadoop QA commented on HBASE-7005:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549566/Hbase-7005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3069//console
This message is automatically generated.
> Upgrade Thrift lib to 0.9.0
> ---
>
> Key: HBASE-7005
> URL: https://issues.apache.org/jira/browse/HBASE-7005
> Project: HBase
> Issue Type: Bug
> Components: Thrift
>Reporter: Jake Farrell
>Priority: Minor
> Attachments: Hbase-7005.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal updated HBASE-6974: - Attachment: HBASE-6974-v4.patch fixing a test failure > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch, HBASE-6974-v4.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal updated HBASE-6974: - Status: Patch Available (was: Open) > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch, HBASE-6974-v4.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal updated HBASE-6974: - Status: Open (was: Patch Available) > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478415#comment-13478415 ] Jarek Jarcec Cecho commented on HBASE-6929: --- Another possible solution would be to provide artifact hbase-test that would have classifier either hadoop1 or hadoop2 based on for which version it was compiled. This solution would be probably step back as for example hadoop was using it in the past and has moved to classifiers, but it should be working without need to have special version for testing artifact compiled against hadoop2. Jarcec > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4435) Add Group By functionality using Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478408#comment-13478408
]
Aaron Tokhy commented on HBASE-4435:
Thanks for the quick review, I'll soon update JIRA with a new patch, based off
of SVN trunk, though not at the moment. Also I'll have to clean up some of the
code, thanks for the quick feedback!
I may also change a few other things, such as using HashedBytes instead of Text
to be able to perform roll-ups of types other than UTF-8 strings.
> Add Group By functionality using Coprocessors
> -
>
> Key: HBASE-4435
> URL: https://issues.apache.org/jira/browse/HBASE-4435
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Nichole Treadway
>Priority: Minor
> Labels: by, coprocessors, group, hbase
> Attachments: HBase-4435.patch, HBASE-4435-v2.patch
>
>
> Adds in a Group By -like functionality to HBase, using the Coprocessor
> framework.
> It provides the ability to group the result set on one or more columns
> (groupBy families). It computes statistics (max, min, sum, count, sum of
> squares, number missing) for a second column, called the stats column.
> To use, I've provided two implementations.
> 1. In the first, you specify a single group-by column and a stats field:
> statsMap = gbc.getStats(tableName, scan, groupByFamily,
> groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The result is a map with the Group By column value (as a String) to a
> GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc.
> of the stats column for that group.
> 2. The second implementation allows you to specify a list of group-by columns
> and a stats field. The List of group-by columns is expected to contain lists
> of {column family, qualifier} pairs.
> statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns,
> statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The GroupByStatsValues code is adapted from the Solr Stats component.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6793) Make hbase-examples module
[ https://issues.apache.org/jira/browse/HBASE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478396#comment-13478396 ] Sergey Shelukhin commented on HBASE-6793: - I'll squash the changes and add final patch here when done with that review. > Make hbase-examples module > -- > > Key: HBASE-6793 > URL: https://issues.apache.org/jira/browse/HBASE-6793 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Enis Soztutar >Assignee: Sergey Shelukhin > Labels: noob > Attachments: HBASE-6793.patch > > > There are some examples under /examples/, which are not compiled as a part of > the build. We can move them to an hbase-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6793) Make hbase-examples module
[ https://issues.apache.org/jira/browse/HBASE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478394#comment-13478394 ] Sergey Shelukhin commented on HBASE-6793: - Updated the review at https://reviews.apache.org/r/7626/, created the jiras > Make hbase-examples module > -- > > Key: HBASE-6793 > URL: https://issues.apache.org/jira/browse/HBASE-6793 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Enis Soztutar >Assignee: Sergey Shelukhin > Labels: noob > Attachments: HBASE-6793.patch > > > There are some examples under /examples/, which are not compiled as a part of > the build. We can move them to an hbase-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7000) Fix the "INT_VACUOUS_COMPARISON" WARNING in KeyValue class
[
https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478391#comment-13478391
]
Ted Yu commented on HBASE-7000:
---
patch looks good.
nit: insert space before 1:
{code}
+ public static final int MAXIMUM_VALUE_LENGTH = Integer.MAX_VALUE -1;
{code}
> Fix the "INT_VACUOUS_COMPARISON" WARNING in KeyValue class
> --
>
> Key: HBASE-7000
> URL: https://issues.apache.org/jira/browse/HBASE-7000
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: liang xie
>Assignee: liang xie
>Priority: Minor
> Attachments: HBASE-7000.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6948) shell create table script cannot handle split key which is expressed in raw bytes
[ https://issues.apache.org/jira/browse/HBASE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6948: -- Fix Version/s: 0.96.0 Intergrated to trunk. Thanks for the review, Stack and Ram. > shell create table script cannot handle split key which is expressed in raw > bytes > - > > Key: HBASE-6948 > URL: https://issues.apache.org/jira/browse/HBASE-6948 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2 >Reporter: Ted Yu >Assignee: Tianying Chang > Fix For: 0.96.0 > > Attachments: HBASE-6948.patch, HBASE-6948-trunk.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6948) shell create table script cannot handle split key which is expressed in raw bytes
[ https://issues.apache.org/jira/browse/HBASE-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-6948: - Assignee: Tianying Chang > shell create table script cannot handle split key which is expressed in raw > bytes > - > > Key: HBASE-6948 > URL: https://issues.apache.org/jira/browse/HBASE-6948 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2 >Reporter: Ted Yu >Assignee: Tianying Chang > Fix For: 0.96.0 > > Attachments: HBASE-6948.patch, HBASE-6948-trunk.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[
https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478369#comment-13478369
]
Hadoop QA commented on HBASE-6965:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549565/OSMXBean_HBASE-6965-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3068//console
This message is automatically generated.
> Generic MXBean Utility class to support all JDK vendors
> ---
>
> Key: HBASE-6965
> URL: https://issues.apache.org/jira/browse/HBASE-6965
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.94.1
>Reporter: Kumar Ravi
>Assignee: Kumar Ravi
> Labels: patch
> Fix For: 0.94.3
>
> Attachments: HBASE-6965.patch, OSMXBean_HBASE-6965-0.94.patch
>
>
> This issue is related to JIRA
> https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to
> propose the use of a newly created generic
> org.apache.hadoop.hbase.util.OSMXBean class that can be used by other
> classes. JIRA HBASE-6945 contains a patch for the class
> org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the
> inclusion of this new class, HBase can be built and become functional with
> JDKs and JREs other than what is provided by Oracle.
> This class uses reflection to determine the JVM vendor (Sun, IBM) and the
> platform (Linux or Windows), and contains other methods that return the OS
> properties - 1. Number of Open File descriptors; 2. Maximum number of File
> Descriptors.
> This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well
> as Oracle JDK 6. Junit tests (runDevTests category) completed without any
> failures or errors when tested on all the three JDKs.The builds and tests
> were attempted on branch hbase-0.94 Revision 1396305.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[ https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7005: -- Status: Patch Available (was: Open) > Upgrade Thrift lib to 0.9.0 > --- > > Key: HBASE-7005 > URL: https://issues.apache.org/jira/browse/HBASE-7005 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jake Farrell >Priority: Minor > Attachments: Hbase-7005.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478354#comment-13478354 ] Roman Shaposhnik commented on HBASE-6929: - So, isn't the lack of secure jars as much a problem as the lack of the ones compiled against different versions of Hadoop? I'm just trying to understand the scope of different 'twists' of HBase that one would have to know/care about. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4435) Add Group By functionality using Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478351#comment-13478351
]
Ted Yu commented on HBASE-4435:
---
I didn't find any test in the patch. It would be difficult for a feature to be
accepted without new tests.
Should GroupByStatsValues be named GroupByStats (since stats imply some values)
?
{code}
+ * Copyright 2012 The Apache Software Foundation
{code}
The above line is no longer needed in license header.
BigDecimalColumnInterpreter is covered in HBASE-6669. To make the workload
reasonable for this JIRA, you can exclude it from patch.
{code}
+public class CharacterColumnInterpreter implements
ColumnInterpreter {
{code}
Add annotation for audience and stability for public classes.
In GroupByClient.java, the following import can be removed:
{code}
+import com.sun.istack.logging.Logger;
{code}
{code}
+Map> getStats(
+ final byte[] tableName, final Scan scan,
+ final List groupByTuples, final byte[][] statsTuple,
{code}
The @param for the above method doesn't match actual parameters - probably you
changed API in later iteration.
{code}
+class RowNumCallback implements
{code}
The above class can be made private.
I think we should find a better name for the above class - it does aggregation.
{code}
+long bt = System.currentTimeMillis();
{code}
Please use EnvironmentEdge instead.
{code}
+table.close();
{code}
Please enclose the above in finally clause.
> Add Group By functionality using Coprocessors
> -
>
> Key: HBASE-4435
> URL: https://issues.apache.org/jira/browse/HBASE-4435
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Nichole Treadway
>Priority: Minor
> Labels: by, coprocessors, group, hbase
> Attachments: HBase-4435.patch, HBASE-4435-v2.patch
>
>
> Adds in a Group By -like functionality to HBase, using the Coprocessor
> framework.
> It provides the ability to group the result set on one or more columns
> (groupBy families). It computes statistics (max, min, sum, count, sum of
> squares, number missing) for a second column, called the stats column.
> To use, I've provided two implementations.
> 1. In the first, you specify a single group-by column and a stats field:
> statsMap = gbc.getStats(tableName, scan, groupByFamily,
> groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The result is a map with the Group By column value (as a String) to a
> GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc.
> of the stats column for that group.
> 2. The second implementation allows you to specify a list of group-by columns
> and a stats field. The List of group-by columns is expected to contain lists
> of {column family, qualifier} pairs.
> statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns,
> statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The GroupByStatsValues code is adapted from the Solr Stats component.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[ https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478339#comment-13478339 ] Todd Lipcon commented on HBASE-7005: Makes sense. Let's do this in trunk only, since the pom dependency change can hurt people's compatibility in earlier versions? > Upgrade Thrift lib to 0.9.0 > --- > > Key: HBASE-7005 > URL: https://issues.apache.org/jira/browse/HBASE-7005 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jake Farrell >Priority: Minor > Attachments: Hbase-7005.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478334#comment-13478334 ] Enis Soztutar commented on HBASE-6929: -- Since classifiers did not work, I could not find another way other than changing the version number for publishing the jars. I think we haven't published secure jars yet. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7005) Upgrade Thrift lib to 0.9.0
[ https://issues.apache.org/jira/browse/HBASE-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jake Farrell updated HBASE-7005: Attachment: Hbase-7005.patch Patch to update Thrift lib from 0.8.0 to latest release 0.9.0 > Upgrade Thrift lib to 0.9.0 > --- > > Key: HBASE-7005 > URL: https://issues.apache.org/jira/browse/HBASE-7005 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Jake Farrell >Priority: Minor > Attachments: Hbase-7005.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[
https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478320#comment-13478320
]
Hadoop QA commented on HBASE-6974:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549554/HBASE-6974-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3067//console
This message is automatically generated.
> Metric for blocked updates
> --
>
> Key: HBASE-6974
> URL: https://issues.apache.org/jira/browse/HBASE-6974
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Michael Drzal
>Priority: Critical
> Fix For: 0.94.3, 0.96.0
>
> Attachments: HBASE-6974.patch, HBASE-6974-v2.patch,
> HBASE-6974-v3.patch
>
>
> When the disc subsystem cannot keep up with a sustained high write load, a
> region will eventually block updates to throttle clients.
> (HRegion.checkResources).
> It would be nice to have a metric for this, so that these occurrences can be
> tracked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7005) Upgrade Thrift lib to 0.9.0
Jake Farrell created HBASE-7005: --- Summary: Upgrade Thrift lib to 0.9.0 Key: HBASE-7005 URL: https://issues.apache.org/jira/browse/HBASE-7005 Project: HBase Issue Type: Bug Components: Thrift Reporter: Jake Farrell Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6965: -- Attachment: OSMXBean_HBASE-6965-0.94.patch Submitting patch for the 0.94.x release. Differences from trunk patch: 1. Path names for OSMXBean.java are different. 2. No changes were required for pom.xml > Generic MXBean Utility class to support all JDK vendors > --- > > Key: HBASE-6965 > URL: https://issues.apache.org/jira/browse/HBASE-6965 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.94.1 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.94.3 > > Attachments: HBASE-6965.patch, OSMXBean_HBASE-6965-0.94.patch > > > This issue is related to JIRA > https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to > propose the use of a newly created generic > org.apache.hadoop.hbase.util.OSMXBean class that can be used by other > classes. JIRA HBASE-6945 contains a patch for the class > org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the > inclusion of this new class, HBase can be built and become functional with > JDKs and JREs other than what is provided by Oracle. > This class uses reflection to determine the JVM vendor (Sun, IBM) and the > platform (Linux or Windows), and contains other methods that return the OS > properties - 1. Number of Open File descriptors; 2. Maximum number of File > Descriptors. > This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well > as Oracle JDK 6. Junit tests (runDevTests category) completed without any > failures or errors when tested on all the three JDKs.The builds and tests > were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4435) Add Group By functionality using Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478311#comment-13478311
]
Ted Yu commented on HBASE-4435:
---
Thanks for the patch.
Can you provide trunk patch following the example of:
HBASE-6785 'Convert AggregateProtocol to protobuf defined coprocessor service'
Will provide comments soon.
For patch of this size, review board (https://reviews.apache.org) would help
reviewers.
> Add Group By functionality using Coprocessors
> -
>
> Key: HBASE-4435
> URL: https://issues.apache.org/jira/browse/HBASE-4435
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Nichole Treadway
>Priority: Minor
> Labels: by, coprocessors, group, hbase
> Attachments: HBase-4435.patch, HBASE-4435-v2.patch
>
>
> Adds in a Group By -like functionality to HBase, using the Coprocessor
> framework.
> It provides the ability to group the result set on one or more columns
> (groupBy families). It computes statistics (max, min, sum, count, sum of
> squares, number missing) for a second column, called the stats column.
> To use, I've provided two implementations.
> 1. In the first, you specify a single group-by column and a stats field:
> statsMap = gbc.getStats(tableName, scan, groupByFamily,
> groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The result is a map with the Group By column value (as a String) to a
> GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc.
> of the stats column for that group.
> 2. The second implementation allows you to specify a list of group-by columns
> and a stats field. The List of group-by columns is expected to contain lists
> of {column family, qualifier} pairs.
> statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns,
> statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The GroupByStatsValues code is adapted from the Solr Stats component.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478309#comment-13478309 ] Roman Shaposhnik commented on HBASE-6929: - Aren't you guys already publishing secure/unsecure bits? Adding a dedicated version would make it 4 then. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478305#comment-13478305 ] Ted Yu commented on HBASE-6965: --- Latest patch looks good. @N: Do you want to take a look ? > Generic MXBean Utility class to support all JDK vendors > --- > > Key: HBASE-6965 > URL: https://issues.apache.org/jira/browse/HBASE-6965 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.94.1 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.94.3 > > Attachments: HBASE-6965.patch > > > This issue is related to JIRA > https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to > propose the use of a newly created generic > org.apache.hadoop.hbase.util.OSMXBean class that can be used by other > classes. JIRA HBASE-6945 contains a patch for the class > org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the > inclusion of this new class, HBase can be built and become functional with > JDKs and JREs other than what is provided by Oracle. > This class uses reflection to determine the JVM vendor (Sun, IBM) and the > platform (Linux or Windows), and contains other methods that return the OS > properties - 1. Number of Open File descriptors; 2. Maximum number of File > Descriptors. > This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well > as Oracle JDK 6. Junit tests (runDevTests category) completed without any > failures or errors when tested on all the three JDKs.The builds and tests > were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5525) Truncate and preserve region boundaries option
[ https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Odell updated HBASE-5525: --- Status: Open (was: Patch Available) > Truncate and preserve region boundaries option > -- > > Key: HBASE-5525 > URL: https://issues.apache.org/jira/browse/HBASE-5525 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jean-Daniel Cryans >Assignee: Kevin Odell > Labels: newbie, noob > Fix For: 0.96.0 > > Attachments: HBASE-5525.patch > > > A tool that would be useful for testing (and maybe in prod too) would be a > truncate option to keep the current region boundaries. Right now what you > have to do is completely kill the table and recreate it with the correct > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5525) Truncate and preserve region boundaries option
[ https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478296#comment-13478296 ] Kevin Odell commented on HBASE-5525: Great feedback I will work on that as soon as I have time. > Truncate and preserve region boundaries option > -- > > Key: HBASE-5525 > URL: https://issues.apache.org/jira/browse/HBASE-5525 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jean-Daniel Cryans >Assignee: Kevin Odell > Labels: newbie, noob > Fix For: 0.96.0 > > Attachments: HBASE-5525.patch > > > A tool that would be useful for testing (and maybe in prod too) would be a > truncate option to keep the current region boundaries. Right now what you > have to do is completely kill the table and recreate it with the correct > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5525) Truncate and preserve region boundaries option
[
https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478289#comment-13478289
]
Ricky Saltzer commented on HBASE-5525:
--
+1 Greg, something like would be great, also updating the docs to reflect what
{noformat}
truncate 'users_table', {PRESERVE => true}
{noformat}
> Truncate and preserve region boundaries option
> --
>
> Key: HBASE-5525
> URL: https://issues.apache.org/jira/browse/HBASE-5525
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Jean-Daniel Cryans
>Assignee: Kevin Odell
> Labels: newbie, noob
> Fix For: 0.96.0
>
> Attachments: HBASE-5525.patch
>
>
> A tool that would be useful for testing (and maybe in prod too) would be a
> truncate option to keep the current region boundaries. Right now what you
> have to do is completely kill the table and recreate it with the correct
> regions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5525) Truncate and preserve region boundaries option
[ https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478283#comment-13478283 ] Ricky Saltzer commented on HBASE-5525: -- Would it make more sense to have one truncate command, and have an optional parameter which preserves the truncate? Rather than having a separate command for truncating and truncating while preserving region keys? > Truncate and preserve region boundaries option > -- > > Key: HBASE-5525 > URL: https://issues.apache.org/jira/browse/HBASE-5525 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jean-Daniel Cryans >Assignee: Kevin Odell > Labels: newbie, noob > Fix For: 0.96.0 > > Attachments: HBASE-5525.patch > > > A tool that would be useful for testing (and maybe in prod too) would be a > truncate option to keep the current region boundaries. Right now what you > have to do is completely kill the table and recreate it with the correct > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[
https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478264#comment-13478264
]
Hadoop QA commented on HBASE-6965:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549540/HBASE-6965.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 9 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3066//console
This message is automatically generated.
> Generic MXBean Utility class to support all JDK vendors
> ---
>
> Key: HBASE-6965
> URL: https://issues.apache.org/jira/browse/HBASE-6965
> Project: HBase
> Issue Type: Improvement
> Components: build
>Affects Versions: 0.94.1
>Reporter: Kumar Ravi
>Assignee: Kumar Ravi
> Labels: patch
> Fix For: 0.94.3
>
> Attachments: HBASE-6965.patch
>
>
> This issue is related to JIRA
> https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to
> propose the use of a newly created generic
> org.apache.hadoop.hbase.util.OSMXBean class that can be used by other
> classes. JIRA HBASE-6945 contains a patch for the class
> org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the
> inclusion of this new class, HBase can be built and become functional with
> JDKs and JREs other than what is provided by Oracle.
> This class uses reflection to determine the JVM vendor (Sun, IBM) and the
> platform (Linux or Windows), and contains other methods that return the OS
> properties - 1. Number of Open File descriptors; 2. Maximum number of File
> Descriptors.
> This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well
> as Oracle JDK 6. Junit tests (runDevTests category) completed without any
> failures or errors when tested on all the three JDKs.The builds and tests
> were attempted on branch hbase-0.94 Revision 1396305.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal updated HBASE-6974: - Status: Open (was: Patch Available) > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal updated HBASE-6974: - Status: Patch Available (was: Open) > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478260#comment-13478260 ] Michael Drzal commented on HBASE-6974: -- Let me know if that works. > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal updated HBASE-6974: - Attachment: HBASE-6974-v3.patch updated moving currentTimeMillis into while loop > Metric for blocked updates > -- > > Key: HBASE-6974 > URL: https://issues.apache.org/jira/browse/HBASE-6974 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Michael Drzal >Priority: Critical > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, > HBASE-6974-v3.patch > > > When the disc subsystem cannot keep up with a sustained high write load, a > region will eventually block updates to throttle clients. > (HRegion.checkResources). > It would be nice to have a metric for this, so that these occurrences can be > tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478252#comment-13478252 ] Lars Hofhansl commented on HBASE-6032: -- I ran the relevant tests. And also used with some other test cases I have. (The set was with a local HDFS, so I didn't observe any performance benefits) > Port HFileBlockIndex improvement from HBASE-5987 > > > Key: HBASE-6032 > URL: https://issues.apache.org/jira/browse/HBASE-6032 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6032.094.txt, 6032-ports-5987.txt, > 6032-ports-5987-v2.txt, 6032v3.txt > > > Excerpt from HBASE-5987: > First, we propose to lookahead for one more block index so that the > HFileScanner would know the start key value of next data block. So if the > target key value for the scan(reSeekTo) is "smaller" than that start kv of > next data block, it means the target key value has a very high possibility in > the current data block (if not in current data block, then the start kv of > next data block should be returned. +Indexing on the start key has some > defects here+) and it shall NOT query the HFileBlockIndex in this case. On > the contrary, if the target key value is "bigger", then it shall query the > HFileBlockIndex. This improvement shall help to reduce the hotness of > HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block > Cache lookup. > This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4435) Add Group By functionality using Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Tokhy updated HBASE-4435:
---
Labels: by coprocessors group hbase (was: )
> Add Group By functionality using Coprocessors
> -
>
> Key: HBASE-4435
> URL: https://issues.apache.org/jira/browse/HBASE-4435
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Nichole Treadway
>Priority: Minor
> Labels: by, coprocessors, group, hbase
> Attachments: HBase-4435.patch, HBASE-4435-v2.patch
>
>
> Adds in a Group By -like functionality to HBase, using the Coprocessor
> framework.
> It provides the ability to group the result set on one or more columns
> (groupBy families). It computes statistics (max, min, sum, count, sum of
> squares, number missing) for a second column, called the stats column.
> To use, I've provided two implementations.
> 1. In the first, you specify a single group-by column and a stats field:
> statsMap = gbc.getStats(tableName, scan, groupByFamily,
> groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The result is a map with the Group By column value (as a String) to a
> GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc.
> of the stats column for that group.
> 2. The second implementation allows you to specify a list of group-by columns
> and a stats field. The List of group-by columns is expected to contain lists
> of {column family, qualifier} pairs.
> statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns,
> statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The GroupByStatsValues code is adapted from the Solr Stats component.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4435) Add Group By functionality using Coprocessors
[
https://issues.apache.org/jira/browse/HBASE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Tokhy updated HBASE-4435:
---
Attachment: HBASE-4435-v2.patch
I have a newer version of the patch:
Improvements:
1) Added implementations of ColumnInterpreter classes so both AggregationClient
and GroupByClient could perform aggregations on Long, Short, Integer, Double,
Float, Character (or unsigned short), and BigDecimal types.
2) The GroupByStatsValues class is a Java generic that constrains on Java types
that only implement the 'Number' interface. This way the generic is
constrained for those types at compile time.
3) Previously, a HashMap was returned at the end of each RPC call. HashMap
uses java.io.Serializable, which is relatively heavyweight. Switched to using
the Hadoop Writable interface so all objects passed between clients and
regionservers use the Hadoop Writable interface.
4) Fixed some validateParameter bugs in the previous patch which would allow
selections of column qualifiers not found in the Scan object to go through.
Caveats:
1) This works well if your resultset fits into memory as group by values are
aggregated into a HashMap on the client. Therefore, if the cardinality of the
aggregation table is too high, you may get an OOME.
2) All aggregations are calculated by the 'GroupByStatsValues' container.
Perhaps at object construction, a 'statsvalues' can be constructed to only
perform some of the aggregations instead of all of them at the same time.
However this operation is Scan (IO) bound, so improvements would be minimal
here.
3) Like all coprocessors that accept a Scan object, if the aggregation is
performing a full table scan, this will run on all regionservers. Each region
level coprocessor is loaded into an IPC handler (default of 10) on the
regionserver. If the regionserver has more regions than IPC handlers, only 10
group by operations will run at a time.
Depending on your table schema, region size and blockCacheHitRatio, your
mileage may vary. If data can be preaggregated for a group by operation, this
patch would be handy for aggregating a single column value projection of the
original full table. A column oriented representation of the original table
would work well in this case, or possibly a client/coprocessor managed
secondary index.
The patch applies cleanly onto HBase 0.92.1 and HBase 0.94.1.
> Add Group By functionality using Coprocessors
> -
>
> Key: HBASE-4435
> URL: https://issues.apache.org/jira/browse/HBASE-4435
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors
>Reporter: Nichole Treadway
>Priority: Minor
> Attachments: HBase-4435.patch, HBASE-4435-v2.patch
>
>
> Adds in a Group By -like functionality to HBase, using the Coprocessor
> framework.
> It provides the ability to group the result set on one or more columns
> (groupBy families). It computes statistics (max, min, sum, count, sum of
> squares, number missing) for a second column, called the stats column.
> To use, I've provided two implementations.
> 1. In the first, you specify a single group-by column and a stats field:
> statsMap = gbc.getStats(tableName, scan, groupByFamily,
> groupByQualifier, statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The result is a map with the Group By column value (as a String) to a
> GroupByStatsValues object. The GroupByStatsValues object has max,min,sum etc.
> of the stats column for that group.
> 2. The second implementation allows you to specify a list of group-by columns
> and a stats field. The List of group-by columns is expected to contain lists
> of {column family, qualifier} pairs.
> statsMap = gbc.getStats(tableName, scan, listOfGroupByColumns,
> statsFamily, statsQualifier, statsFieldColumnInterpreter);
> The GroupByStatsValues code is adapted from the Solr Stats component.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5525) Truncate and preserve region boundaries option
[ https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478248#comment-13478248 ] Gregory Chanan commented on HBASE-5525: --- Looks pretty good. Maybe truncate_preservebounds instead of truncate_preserve (so it is clear what you are preserving)? Or make preserve a parameter to truncate, default false? (the later would be my personal preference). Also, you could avoid duplicating some code if you had truncate and truncate_preserve call a common function (or just have preserve be a parameter to truncate, as above). > Truncate and preserve region boundaries option > -- > > Key: HBASE-5525 > URL: https://issues.apache.org/jira/browse/HBASE-5525 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.96.0 >Reporter: Jean-Daniel Cryans >Assignee: Kevin Odell > Labels: newbie, noob > Fix For: 0.96.0 > > Attachments: HBASE-5525.patch > > > A tool that would be useful for testing (and maybe in prod too) would be a > truncate option to keep the current region boundaries. Right now what you > have to do is completely kill the table and recreate it with the correct > regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6979) recovered.edits file should not break distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478243#comment-13478243 ] Hudson commented on HBASE-6979: --- Integrated in HBase-TRUNK #3452 (See [https://builds.apache.org/job/HBase-TRUNK/3452/]) HBASE-6979 recovered.edits file should not break distributed log splitting (Revision 1399352) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java > recovered.edits file should not break distributed log splitting > --- > > Key: HBASE-6979 > URL: https://issues.apache.org/jira/browse/HBASE-6979 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: trunk-6979.patch > > > Distributed log splitting fails in creating the recovered.edits folder during > upgrade because there is a file called recovered.edits there. > Instead of checking if the patch exists, we need to check if it exists and is > a path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6980) Parallel Flushing Of Memstores
[ https://issues.apache.org/jira/browse/HBASE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478212#comment-13478212 ] Todd Lipcon commented on HBASE-6980: If I remember correctly, there is a reason for the flush marker: it ensures that the RS hasn't been fenced on HDFS -- i.e that it hasn't lost its connection to ZK and already had its log splitting started. The reason this is important is that, otherwise, it could move on to delete old log segments, which would potentially break the log split process. It may be that the locking can be more lax, though. > Parallel Flushing Of Memstores > -- > > Key: HBASE-6980 > URL: https://issues.apache.org/jira/browse/HBASE-6980 > Project: HBase > Issue Type: New Feature >Reporter: Kannan Muthukkaruppan >Assignee: Kannan Muthukkaruppan > > For write dominated workloads, single threaded memstore flushing is an > unnecessary bottleneck. With a single flusher thread, we are basically not > setup to take advantage of the aggregate throughput that multi-disk nodes > provide. > * For puts with WAL enabled, the bottleneck is more likely the "single" WAL > per region server. So this particular fix may not buy as much unless we > unlock that bottleneck with multiple commit logs per region server. (Topic > for a separate JIRA-- HBASE-6981). > * But for puts with WAL disabled (e.g., when using HBASE-5783 style fast bulk > imports), we should be able to support much better ingest rates with parallel > flushing of memstores. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6651) Thread safety of HTablePool is doubtful
[
https://issues.apache.org/jira/browse/HBASE-6651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478198#comment-13478198
]
Hadoop QA commented on HBASE-6651:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549457/HBASE-6651-V2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
83 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3065//console
This message is automatically generated.
> Thread safety of HTablePool is doubtful
> ---
>
> Key: HBASE-6651
> URL: https://issues.apache.org/jira/browse/HBASE-6651
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.94.1
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-6651.patch, HBASE-6651-V2.patch, sample.zip,
> sample.zip, sharedmap_for_hbaseclient.zip
>
>
> There are some operations in HTablePool to access to PoolMap in multiple
> times without any explict synchronization.
> For example HTablePool.closeTablePool() calles PoolMap.values(), and calles
> PoolMap.remove(). If other threads add new instances to the pool in the
> middle of the calls, the new added instances might be dropped.
> (HTablePool.closeTablePool() also has another problem that calling it by
> multple threads causes accessing HTable by multiple threads.)
> Moreover, PoolMap is not thread safe for the same reason.
> For example PoolMap.put() calles ConcurrentMap.get() and calles
> ConcurrentMap.put(). If other threads add a new instance to the concurent map
> in the middle of the calls, the new instance might be dropped.
> And also implementations of Pool have the same problems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6965: -- Status: Open (was: Patch Available) > Generic MXBean Utility class to support all JDK vendors > --- > > Key: HBASE-6965 > URL: https://issues.apache.org/jira/browse/HBASE-6965 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.94.1 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.94.3 > > Attachments: HBASE-6965.patch > > > This issue is related to JIRA > https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to > propose the use of a newly created generic > org.apache.hadoop.hbase.util.OSMXBean class that can be used by other > classes. JIRA HBASE-6945 contains a patch for the class > org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the > inclusion of this new class, HBase can be built and become functional with > JDKs and JREs other than what is provided by Oracle. > This class uses reflection to determine the JVM vendor (Sun, IBM) and the > platform (Linux or Windows), and contains other methods that return the OS > properties - 1. Number of Open File descriptors; 2. Maximum number of File > Descriptors. > This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well > as Oracle JDK 6. Junit tests (runDevTests category) completed without any > failures or errors when tested on all the three JDKs.The builds and tests > were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6965: -- Status: Patch Available (was: Open) > Generic MXBean Utility class to support all JDK vendors > --- > > Key: HBASE-6965 > URL: https://issues.apache.org/jira/browse/HBASE-6965 > Project: HBase > Issue Type: Improvement > Components: build >Affects Versions: 0.94.1 >Reporter: Kumar Ravi >Assignee: Kumar Ravi > Labels: patch > Fix For: 0.94.3 > > Attachments: HBASE-6965.patch > > > This issue is related to JIRA > https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to > propose the use of a newly created generic > org.apache.hadoop.hbase.util.OSMXBean class that can be used by other > classes. JIRA HBASE-6945 contains a patch for the class > org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the > inclusion of this new class, HBase can be built and become functional with > JDKs and JREs other than what is provided by Oracle. > This class uses reflection to determine the JVM vendor (Sun, IBM) and the > platform (Linux or Windows), and contains other methods that return the OS > properties - 1. Number of Open File descriptors; 2. Maximum number of File > Descriptors. > This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well > as Oracle JDK 6. Junit tests (runDevTests category) completed without any > failures or errors when tested on all the three JDKs.The builds and tests > were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
