[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478714#comment-13478714 ] Devaraj Das commented on HBASE-6758: This should mostly be applicable on 0.94 straightaway.. [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7008) Set scanner caching to a better default
liang xie created HBASE-7008: Summary: Set scanner caching to a better default Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Attachment: HBASE-7008.patch Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Description: per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liang xie updated HBASE-7008: - Assignee: liang xie Status: Patch Available (was: Open) Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Assignee: liang xie Attachments: HBASE-7008.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-6786: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, DD. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478732#comment-13478732 ] Hudson commented on HBASE-6758: --- Integrated in HBase-TRUNK #3455 (See [https://builds.apache.org/job/HBase-TRUNK/3455/]) HBASE-6758 [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file (Devaraj Das via JD) (Revision 1399517) Result = FAILURE jdcryans : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7002) Fix all 4 findbug performance warnings
[ https://issues.apache.org/jira/browse/HBASE-7002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478733#comment-13478733 ] Hudson commented on HBASE-7002: --- Integrated in HBase-TRUNK #3455 (See [https://builds.apache.org/job/HBase-TRUNK/3455/]) HBASE-7002 Fix all 4 findbug performance warnings (Liang Xie) (Revision 1399514) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java Fix all 4 findbug performance warnings -- Key: HBASE-7002 URL: https://issues.apache.org/jira/browse/HBASE-7002 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7002.patch Fix the perf warning from this report : https://builds.apache.org/job/PreCommit-HBASE-Build/3057//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478736#comment-13478736
]
Hadoop QA commented on HBASE-5498:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549633/HBASE-5498_trunk_3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
83 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3074//console
This message is automatically generated.
Secure Bulk Load
Key: HBASE-5498
URL: https://issues.apache.org/jira/browse/HBASE-5498
Project: HBase
Issue Type: Improvement
Components: security
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.94.3, 0.96.0
Attachments: HBASE-5498_94_2.patch, HBASE-5498_94_3.patch,
HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch,
HBASE-5498_draft.patch, HBASE-5498_trunk_2.patch, HBASE-5498_trunk_3.patch,
HBASE-5498_trunk.patch
Design doc:
https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
Short summary:
Security as it stands does not cover the bulkLoadHFiles() feature. Users
calling this method will bypass ACLs. Also loading is made more cumbersome in
a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
from user's directory to the hbase directory, which would require certain
write access privileges set.
Our solution is to create a coprocessor which makes use of AuthManager to
verify if a user has write access to the table. If so, launches a MR job as
the hbase user to do the importing (ie rewrite from text to hfiles). One
tricky part this job will have to do is impersonate the calling user when
reading the input files. We can do this by expecting the user to pass an hdfs
delegation token as part of the secureBulkLoad() coprocessor call and extend
an inputformat to make use of that token. The output is written to a
temporary directory accessible only by hbase and then bulkloadHFiles() is
called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478769#comment-13478769 ] Ted Yu commented on HBASE-6786: --- Looks like Mutation was spelled as Mutatation in some method names Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7006) [MTTR] Study distributed log splitting to see how we can make it faster
[ https://issues.apache.org/jira/browse/HBASE-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478770#comment-13478770 ] nkeywal commented on HBASE-7006: Nothing related to HBASE-6738? There is not a limit of 32 WALs per node (hence 900 wals)? Or have you lost more nodes? [MTTR] Study distributed log splitting to see how we can make it faster --- Key: HBASE-7006 URL: https://issues.apache.org/jira/browse/HBASE-7006 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Fix For: 0.96.0 Just saw interesting issue where a cluster went down hard and 30 nodes had 1700 WALs to replay. Replay took almost an hour. It looks like it could run faster that much of the time is spent zk'ing and nn'ing. Putting in 0.96 so it gets a look at least. Can always punt. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7007) [MTTR] Study assigns to see if we can make them faster still
[ https://issues.apache.org/jira/browse/HBASE-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478772#comment-13478772 ] nkeywal commented on HBASE-7007: That what my tests were saying in June. Hopefully they're right ;-). I've not redone them recently. Hopefully it's still there. (from HBASE-5843) I tested the following scenarios, on a local machine, a pseudo distributed cluster with ZooKeeper and HBase writing in a ram drive, no datanode nor namenode, with 2 region servers, and one empty table with 1 regions, 5K on each RS. Versions taken monday 2nd 3) Start of the cluster after a clean stop; wait for all regions to become online. 0.92: ~1020s 0.94: ~1023s (tested once only) 0.96: ~31s [MTTR] Study assigns to see if we can make them faster still Key: HBASE-7007 URL: https://issues.apache.org/jira/browse/HBASE-7007 Project: HBase Issue Type: Improvement Reporter: stack Looking at a cluster start, I saw that it took about 25 minutes to assign and open 17k regions. 8 minutes was bulk assigning via zk. 17 minutes was opening the regions. HBASE-6640 [0.89-fb] Allow multiple regions to be opened simultaneously in trunk would help but maybe we can do less work up in zk for instance; e.g. if 3.4.5, we can do bulk ops in zk (or make a bulk assign znode that has many regions instead of one) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478777#comment-13478777 ] Hudson commented on HBASE-6032: --- Integrated in HBase-0.94 #540 (See [https://builds.apache.org/job/HBase-0.94/540/]) HBASE-6032 Addendum, forgot to add two files (Revision 1399521) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockWithScanInfo.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksScanned.java Port HFileBlockIndex improvement from HBASE-5987 Key: HBASE-6032 URL: https://issues.apache.org/jira/browse/HBASE-6032 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.94.3, 0.96.0 Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt Excerpt from HBASE-5987: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7008) Set scanner caching to a better default
[
https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478784#comment-13478784
]
Hadoop QA commented on HBASE-7008:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549645/HBASE-7008.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3075//console
This message is automatically generated.
Set scanner caching to a better default
---
Key: HBASE-7008
URL: https://issues.apache.org/jira/browse/HBASE-7008
Project: HBase
Issue Type: Bug
Components: Client
Reporter: liang xie
Assignee: liang xie
Attachments: HBASE-7008.patch
per
http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+
let's set to 100 by default
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478794#comment-13478794 ] Hudson commented on HBASE-6786: --- Integrated in HBase-TRUNK #3456 (See [https://builds.apache.org/job/HBase-TRUNK/3456/]) HBASE-6786 Convert MultiRowMutationProtocol to protocol buffer service (Devaraj Das) (Revision 1399529) Result = FAILURE garyh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MultiRowMutation.java * /hbase/trunk/hbase-server/src/main/protobuf/MultiRowMutation.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478913#comment-13478913 ] Michael Drzal commented on HBASE-6974: -- I hate naming things, and I really don't care what they are named. I did think having HighWater in the name was useful as that term is used in other places in programming/computer science and is generally well understood. Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6611) Forcing region state offline cause double assignment
[ https://issues.apache.org/jira/browse/HBASE-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478916#comment-13478916 ] ramkrishna.s.vasudevan commented on HBASE-6611: --- @Jimmy Comments on review board. Mainly wrt to some scenarios that are likely to happen. Forcing region state offline cause double assignment Key: HBASE-6611 URL: https://issues.apache.org/jira/browse/HBASE-6611 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: trunk-6611_v2.patch In assigning a region, assignment manager forces the region state offline if it is not. This could cause double assignment, for example, if the region is already assigned and in the Open state, you should not just change it's state to Offline, and assign it again. I think this could be the root cause for all double assignments IF the region state is reliable. After this loophole is closed, TestHBaseFsck should come up a different way to create some assignment inconsistencies, for example, calling region server to open a region directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478926#comment-13478926 ] Hudson commented on HBASE-6786: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #226 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/226/]) HBASE-6786 Convert MultiRowMutationProtocol to protocol buffer service (Devaraj Das) (Revision 1399529) Result = FAILURE garyh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationProtocol.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MultiRowMutation.java * /hbase/trunk/hbase-server/src/main/protobuf/MultiRowMutation.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478925#comment-13478925 ] Hudson commented on HBASE-6758: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #226 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/226/]) HBASE-6758 [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file (Devaraj Das via JD) (Revision 1399517) Result = FAILURE jdcryans : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6611) Forcing region state offline cause double assignment
[ https://issues.apache.org/jira/browse/HBASE-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13478944#comment-13478944 ] ramkrishna.s.vasudevan commented on HBASE-6611: --- Now i am also getting in sync with AM code in trunk..Thanks to Jimmy for making it more reliable. Forcing region state offline cause double assignment Key: HBASE-6611 URL: https://issues.apache.org/jira/browse/HBASE-6611 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: trunk-6611_v2.patch In assigning a region, assignment manager forces the region state offline if it is not. This could cause double assignment, for example, if the region is already assigned and in the Open state, you should not just change it's state to Offline, and assign it again. I think this could be the root cause for all double assignments IF the region state is reliable. After this loophole is closed, TestHBaseFsck should come up a different way to create some assignment inconsistencies, for example, calling region server to open a region directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kishore updated HBASE-6389:
--
Attachment: HBASE-6389_trunk_v2.patch
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-6786: --- Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kishore updated HBASE-6389:
--
Status: Patch Available (was: Open)
The new patch addresses test failures in couple of test cases (including
TestReplication, TestZookeeper).
Also, since the config params are used in quite a few places, I have moved them
as a public constant.
Please note that there is no change in the logic of the fix. This only fixes
few bugs in test cases which surfaced due to the fix.
I'll submit the review request later during the day.
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it
[jira] [Commented] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479058#comment-13479058
]
Hadoop QA commented on HBASE-6389:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549692/HBASE-6389_trunk_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3076//console
This message is automatically generated.
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
[jira] [Updated] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6786: --- Attachment: 6786-1-addendum.patch Oops. I had misspelt mutation in the .proto file. Attached is the corrected .proto file (this patch is on top of what got committed). Thanks for spotting this, [~te...@apache.org]. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6965: -- Fix Version/s: 0.96.0 Generic MXBean Utility class to support all JDK vendors --- Key: HBASE-6965 URL: https://issues.apache.org/jira/browse/HBASE-6965 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.1 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6965.patch, OSMXBean_HBASE-6965-0.94.patch This issue is related to JIRA https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to propose the use of a newly created generic org.apache.hadoop.hbase.util.OSMXBean class that can be used by other classes. JIRA HBASE-6945 contains a patch for the class org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the inclusion of this new class, HBase can be built and become functional with JDKs and JREs other than what is provided by Oracle. This class uses reflection to determine the JVM vendor (Sun, IBM) and the platform (Linux or Windows), and contains other methods that return the OS properties - 1. Number of Open File descriptors; 2. Maximum number of File Descriptors. This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well as Oracle JDK 6. Junit tests (runDevTests category) completed without any failures or errors when tested on all the three JDKs.The builds and tests were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479134#comment-13479134 ] Ted Yu commented on HBASE-6965: --- Integrated to trunk. Thanks for the patch, Kumar. @Lars: Do you think this should go into 0.94 as well ? Generic MXBean Utility class to support all JDK vendors --- Key: HBASE-6965 URL: https://issues.apache.org/jira/browse/HBASE-6965 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.1 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6965.patch, OSMXBean_HBASE-6965-0.94.patch This issue is related to JIRA https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to propose the use of a newly created generic org.apache.hadoop.hbase.util.OSMXBean class that can be used by other classes. JIRA HBASE-6945 contains a patch for the class org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the inclusion of this new class, HBase can be built and become functional with JDKs and JREs other than what is provided by Oracle. This class uses reflection to determine the JVM vendor (Sun, IBM) and the platform (Linux or Windows), and contains other methods that return the OS properties - 1. Number of Open File descriptors; 2. Maximum number of File Descriptors. This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well as Oracle JDK 6. Junit tests (runDevTests category) completed without any failures or errors when tested on all the three JDKs.The builds and tests were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479138#comment-13479138 ] Lars Hofhansl commented on HBASE-6974: -- Same here. Seems in each patch that is the hardest part :) How about updatesBlockedSeconds and updatesBlockedSecondsHighWater? Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7009: Description: Need to port. I am porting V5 patch from the original JIRA; I have partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface (was: Need to port. I am porting V5 patch from this item; I have partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface) Port HBaseCluster interface/tests to 0.94 - Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Need to port. I am porting V5 patch from the original JIRA; I have partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479139#comment-13479139 ] Gary Helmling commented on HBASE-6786: -- Nice catch, Ted. Thanks! I'm testing the addendum now. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
Sergey Shelukhin created HBASE-7009: --- Summary: Port HBaseCluster interface/tests to 0.94 Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Need to port. I am porting V5 patch from this item; I have partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7009: Description: Need to port. I am porting V5 patch from the original JIRA; I have a partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface (was: Need to port. I am porting V5 patch from the original JIRA; I have partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface) Port HBaseCluster interface/tests to 0.94 - Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Need to port. I am porting V5 patch from the original JIRA; I have a partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Attachment: (was: ResourceCheckerJunitListener_HBASE_6945-trunk.patch) Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceChecker-0.94.patch, ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Attachment: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceChecker-0.94.patch, ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Attachment: ResourceChecker-0.94.patch Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceChecker-0.94.patch, ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Status: Patch Available (was: Open) Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceChecker-0.94.patch, ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479156#comment-13479156 ] Gary Helmling commented on HBASE-6786: -- +1 on addendum. [~ted_yu] do you have any comments on the addendum, since you spotted the problem? Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479157#comment-13479157 ] Ted Yu commented on HBASE-7000: --- The two failed tests are flaky. Integrated to trunk. Thanks for the patch Liang. Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7000.patch, HBASE-7000-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7000: -- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7000.patch, HBASE-7000-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[
https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479159#comment-13479159
]
Hadoop QA commented on HBASE-6945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549711/ResourceChecker-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3077//console
This message is automatically generated.
Compilation errors when using non-Sun JDKs to build HBase-0.94
--
Key: HBASE-6945
URL: https://issues.apache.org/jira/browse/HBASE-6945
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.94.1
Environment: RHEL 6.3, IBM Java 7
Reporter: Kumar Ravi
Assignee: Kumar Ravi
Labels: patch
Fix For: 0.94.3
Attachments: ResourceChecker-0.94.patch,
ResourceCheckerJUnitListener_HBASE_6945-trunk.patch
When using IBM Java 7 to build HBase-0.94.1, the following comilation error
is seen.
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25]
error: package com.sun.management does not exist
[ERROR]
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25]
error: cannot find symbol
[ERROR] symbol: class UnixOperatingSystemMXBean
location: class ResourceAnalyzer
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29]
error: cannot find symbol
[ERROR] symbol: class UnixOperatingSystemMXBean
location: class ResourceAnalyzer
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23]
error: cannot find symbol
[INFO] 4 errors
[INFO] -
[INFO]
[INFO] BUILD FAILURE
[INFO]
I have a patch available which should work for all JDKs including Sun.
I am in the process of testing this patch. Preliminary tests indicate the
build is working fine with this patch. I will post this patch when I am done
testing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479160#comment-13479160 ] Ted Yu commented on HBASE-6786: --- Thanks for the quick responses, Devaraj and Gary. We have MultiRowMutationService with two messages, MultiMutateRequest and MultiMutateResponse. I think it would be better if the verb in messages is changed to noun, Mutation. My two cents. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479163#comment-13479163 ] Ted Yu commented on HBASE-6945: --- @Kumar: Hadoop QA is not smart enough to pick the trunk patch among the two patches you just uploaded. You can upload trunk patch again for QA to run test suite. Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceChecker-0.94.patch, ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6398) Print a warning if there is no local datanode
[ https://issues.apache.org/jira/browse/HBASE-6398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479172#comment-13479172 ] Sameer Vaishampayan commented on HBASE-6398: Don't know why St.Ack's comments are not here. I remember his response was that the patch as is will get report info for all data nodes every time the method is called, therefore a performance issue. This I am trying to rectify by adding a new API which will return me a data node info for a particular data node given a hostname. This change is in Hadoop hdfs and therefore has a longer dependency. The review for Hadoop changes is out : https://reviews.apache.org/r/7630/ Print a warning if there is no local datanode - Key: HBASE-6398 URL: https://issues.apache.org/jira/browse/HBASE-6398 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Sameer Vaishampayan Labels: noob Attachments: 6398-4.patch When starting up a RS HBase should print out a warning if there is no datanode locally. Lots of optimizations are only available if the data is machine local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479170#comment-13479170 ] Devaraj Das commented on HBASE-6786: FWIW, I'd prefer to leave the message names as is. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479174#comment-13479174 ] Gary Helmling commented on HBASE-6786: -- While Mutate doesn't read quite right to me either, I think we should leave it as is, so that it's consistent with the existing Mutate, MutateRequest, and MutateResponse in Client.proto. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479179#comment-13479179 ] Ted Yu commented on HBASE-6786: --- That was where such names came from :-) +1 on addendum. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479182#comment-13479182 ] Lars Hofhansl commented on HBASE-6032: -- Ran queueFailover manually. Passes locally. Port HFileBlockIndex improvement from HBASE-5987 Key: HBASE-6032 URL: https://issues.apache.org/jira/browse/HBASE-6032 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.94.3, 0.96.0 Attachments: 6032.094.txt, 6032-ports-5987.txt, 6032-ports-5987-v2.txt, 6032v3.txt Excerpt from HBASE-5987: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479188#comment-13479188 ] Hudson commented on HBASE-6965: --- Integrated in HBase-TRUNK #3457 (See [https://builds.apache.org/job/HBase-TRUNK/3457/]) HBASE-6965 Generic MXBean Utility class to support all JDK vendors (Kumar Ravi) (Revision 1399734) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/OSMXBean.java Generic MXBean Utility class to support all JDK vendors --- Key: HBASE-6965 URL: https://issues.apache.org/jira/browse/HBASE-6965 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.1 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6965.patch, OSMXBean_HBASE-6965-0.94.patch This issue is related to JIRA https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to propose the use of a newly created generic org.apache.hadoop.hbase.util.OSMXBean class that can be used by other classes. JIRA HBASE-6945 contains a patch for the class org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the inclusion of this new class, HBase can be built and become functional with JDKs and JREs other than what is provided by Oracle. This class uses reflection to determine the JVM vendor (Sun, IBM) and the platform (Linux or Windows), and contains other methods that return the OS properties - 1. Number of Open File descriptors; 2. Maximum number of File Descriptors. This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well as Oracle JDK 6. Junit tests (runDevTests category) completed without any failures or errors when tested on all the three JDKs.The builds and tests were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479192#comment-13479192 ] Michael Drzal commented on HBASE-6974: -- Sure, that is fine. Do you want me to spin a new patch? Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling resolved HBASE-6786. -- Resolution: Fixed Thanks Ted and DD. Addendum committed to trunk. Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6793) Make hbase-examples module
[ https://issues.apache.org/jira/browse/HBASE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-6793: Attachment: HBASE-6793-v2.patch Attaching full patch so that the status would change... Make hbase-examples module -- Key: HBASE-6793 URL: https://issues.apache.org/jira/browse/HBASE-6793 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Enis Soztutar Assignee: Sergey Shelukhin Labels: noob Attachments: HBASE-6793.patch, HBASE-6793-v2.patch There are some examples under /examples/, which are not compiled as a part of the build. We can move them to an hbase-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6793) Make hbase-examples module
[ https://issues.apache.org/jira/browse/HBASE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-6793: Status: Patch Available (was: Open) Make hbase-examples module -- Key: HBASE-6793 URL: https://issues.apache.org/jira/browse/HBASE-6793 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Enis Soztutar Assignee: Sergey Shelukhin Labels: noob Attachments: HBASE-6793.patch, HBASE-6793-v2.patch There are some examples under /examples/, which are not compiled as a part of the build. We can move them to an hbase-examples module. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479229#comment-13479229 ] Lars Hofhansl commented on HBASE-6974: -- Only if you want to, I can make these changes upon commmit. Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6786) Convert MultiRowMutationProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479238#comment-13479238 ] Hudson commented on HBASE-6786: --- Integrated in HBase-TRUNK #3458 (See [https://builds.apache.org/job/HBase-TRUNK/3458/]) HBASE-6786 addendum - fix mutation spelling in .proto and generated classes (Revision 1399755) Result = FAILURE garyh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/MultiRowMutationEndpoint.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MultiRowMutation.java * /hbase/trunk/hbase-server/src/main/protobuf/MultiRowMutation.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java Convert MultiRowMutationProtocol to protocol buffer service --- Key: HBASE-6786 URL: https://issues.apache.org/jira/browse/HBASE-6786 Project: HBase Issue Type: Sub-task Components: Coprocessors Reporter: Gary Helmling Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6786-1-addendum.patch, 6786-1.patch With coprocessor endpoints now exposed as protobuf defined services, we should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7000) Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class
[ https://issues.apache.org/jira/browse/HBASE-7000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479237#comment-13479237 ] Hudson commented on HBASE-7000: --- Integrated in HBase-TRUNK #3458 (See [https://builds.apache.org/job/HBase-TRUNK/3458/]) HBASE-7000 Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class (Liang Xie) (Revision 1399740) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java Fix the INT_VACUOUS_COMPARISON WARNING in KeyValue class -- Key: HBASE-7000 URL: https://issues.apache.org/jira/browse/HBASE-7000 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: liang xie Assignee: liang xie Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7000.patch, HBASE-7000-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7009: Attachment: HBASE-7009.patch V1. Haven't run actual integration tests yet. Port HBaseCluster interface/tests to 0.94 - Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7009.patch Need to port. I am porting V5 patch from the original JIRA; I have a partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7009: Status: Patch Available (was: Open) Port HBaseCluster interface/tests to 0.94 - Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7009.patch Need to port. I am porting V5 patch from the original JIRA; I have a partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6981) multiple commit logs per region server
[ https://issues.apache.org/jira/browse/HBASE-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kannan Muthukkaruppan updated HBASE-6981: - Assignee: Liyin Tang (was: Kannan Muthukkaruppan) multiple commit logs per region server -- Key: HBASE-6981 URL: https://issues.apache.org/jira/browse/HBASE-6981 Project: HBase Issue Type: New Feature Reporter: Kannan Muthukkaruppan Assignee: Liyin Tang For write dominated workloads, the fact that we have only 1 commit log per server artificially limits the aggregate throughput to that of one disk (this is especially true if we do true FS level syncs on each datanode, or even frequent intermittent FS level syncs as data is being written to each block). We could consider allowing a configurable number of commit logs per server (perhaps something close to or slightly less than the number of disks per server), and we could shard regions on the server, by maybe just a simple modulo scheme, into those commit logs. [A quick way to experiment with the same might be to run multiple region server instances per server; but might be better operationally to package the feature into a single region server.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6793) Make hbase-examples module
[
https://issues.apache.org/jira/browse/HBASE-6793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479277#comment-13479277
]
Hadoop QA commented on HBASE-6793:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549724/HBASE-6793-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 11 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3078//console
This message is automatically generated.
Make hbase-examples module
--
Key: HBASE-6793
URL: https://issues.apache.org/jira/browse/HBASE-6793
Project: HBase
Issue Type: Improvement
Affects Versions: 0.96.0
Reporter: Enis Soztutar
Assignee: Sergey Shelukhin
Labels: noob
Attachments: HBASE-6793.patch, HBASE-6793-v2.patch
There are some examples under /examples/, which are not compiled as a part of
the build. We can move them to an hbase-examples module.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6577) RegionScannerImpl.nextRow() should seek to next row
[ https://issues.apache.org/jira/browse/HBASE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479282#comment-13479282 ] Jean-Daniel Cryans commented on HBASE-6577: --- Ah sorry I totally missed that you posted a new patch. So I tried to remember exactly how we got the issue (the exact cluster, which table was doing that, etc) and it's fuzzy for both [~eclark] and I. This week wouldn't definitely not be a good week to break things tho, so I currently don't feel like deploying it in the wild. I can try to see if it's possible put it on a RS that has a copy of the data and run those filters on it but I don't have time for that right now. RegionScannerImpl.nextRow() should seek to next row --- Key: HBASE-6577 URL: https://issues.apache.org/jira/browse/HBASE-6577 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0 Attachments: 6577-0.94.txt, 6577.txt, 6577-v2.txt, 6577-v3.txt RegionScannerImpl.nextRow() is called when a filter filters the entire row. In that case we should seek to the next row rather then iterating over all versions of all columns to get there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6951) Allow the master info server to be started in a read only mode.
[ https://issues.apache.org/jira/browse/HBASE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6951: - Attachment: HBASE-6951-094-0.patch HBASE-6951-092-0.patch 94 and 92 versions of the patch Allow the master info server to be started in a read only mode. --- Key: HBASE-6951 URL: https://issues.apache.org/jira/browse/HBASE-6951 Project: HBase Issue Type: Improvement Components: UI Reporter: Elliott Clark Assignee: Elliott Clark Priority: Critical Labels: noob Attachments: HBASE-6951-092-0.patch, HBASE-6951-094-0.patch, HBASE-6951-trunk-0.patch There are some cases that a user could want a web ui to be accessible but might not want the split and compact functionality to be usable. Allowing the web ui to start in a readOnly mode would be good. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479299#comment-13479299
]
Hadoop QA commented on HBASE-7009:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549733/HBASE-7009.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 70 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3079//console
This message is automatically generated.
Port HBaseCluster interface/tests to 0.94
-
Key: HBASE-7009
URL: https://issues.apache.org/jira/browse/HBASE-7009
Project: HBase
Issue Type: Sub-task
Components: test
Affects Versions: 0.94.3
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HBASE-7009.patch
Need to port. I am porting V5 patch from the original JIRA; I have a
partially ported (V3) patch from Enis with protocol buffers being reverted to
HRegionInterface/HMasterInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6951) Allow the master info server to be started in a read only mode.
[
https://issues.apache.org/jira/browse/HBASE-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479301#comment-13479301
]
Hadoop QA commented on HBASE-6951:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549744/HBASE-6951-094-0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3080//console
This message is automatically generated.
Allow the master info server to be started in a read only mode.
---
Key: HBASE-6951
URL: https://issues.apache.org/jira/browse/HBASE-6951
Project: HBase
Issue Type: Improvement
Components: UI
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Critical
Labels: noob
Attachments: HBASE-6951-092-0.patch, HBASE-6951-094-0.patch,
HBASE-6951-trunk-0.patch
There are some cases that a user could want a web ui to be accessible but
might not want the split and compact functionality to be usable.
Allowing the web ui to start in a readOnly mode would be good.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6577) RegionScannerImpl.nextRow() should seek to next row
[
https://issues.apache.org/jira/browse/HBASE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479302#comment-13479302
]
Lars Hofhansl commented on HBASE-6577:
--
I think you had just deployed it and because of the way the thrift uses filters
you ran into this issue. You said this on the mailing list:
{code}
We use thrift's scanOpenWithPrefix which does this:
Filter f = new WhileMatchFilter(
new PrefixFilter(getBytes(startAndPrefix)));
So that might just be it.
{code}
No hurry, I am too scared of this to want to commit this any time soon :)
(I did a lot of testing beforehand and I did not see the issue you reported).
RegionScannerImpl.nextRow() should seek to next row
---
Key: HBASE-6577
URL: https://issues.apache.org/jira/browse/HBASE-6577
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.96.0
Attachments: 6577-0.94.txt, 6577.txt, 6577-v2.txt, 6577-v3.txt
RegionScannerImpl.nextRow() is called when a filter filters the entire row.
In that case we should seek to the next row rather then iterating over all
versions of all columns to get there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7010) PrefixFilter should seek to first matching row
Lars Hofhansl created HBASE-7010: Summary: PrefixFilter should seek to first matching row Key: HBASE-7010 URL: https://issues.apache.org/jira/browse/HBASE-7010 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.3, 0.96.0 Current a PrefixFilter will happily scan all KVs prefix. If should seek forward to the prefix if the current KV prefix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7010) PrefixFilter should seek to first matching row
[ https://issues.apache.org/jira/browse/HBASE-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7010: - Description: Currently a PrefixFilter will happily scan all KVs prefix. If should seek forward to the prefix if the current KV prefix. was: Current a PrefixFilter will happily scan all KVs prefix. If should seek forward to the prefix if the current KV prefix. PrefixFilter should seek to first matching row -- Key: HBASE-7010 URL: https://issues.apache.org/jira/browse/HBASE-7010 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.3, 0.96.0 Currently a PrefixFilter will happily scan all KVs prefix. If should seek forward to the prefix if the current KV prefix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect
[
https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479326#comment-13479326
]
stack commented on HBASE-5974:
--
Thanks for reminding me about this important issue Anoop. Lets get it into
trunk at least (Should be a blocker on 0.96).
Minor: Do you foresee the sequencing being used other than in scanners (Its
implemented in ScannerCallable only at moment)? If NOT, would
OutOfOrderNextException or OutOfOrderScannerNextException be a better name than
CallSequenceOutOfOrderException? If you do this, would suggest changing param
names from callSeq to nextSeq? The former is generic.
Is CallSequenceOutOfOrderException an exception that will bubble up into the
application or is the client its intended audience? If so, should we annotate
it @InterfaceAudience.Private as Abortable is for instance?
{code}
+ if ((cause == null || (!(cause instanceof
NotServingRegionException)
+ !(cause instanceof RegionServerStoppedException)))
+ !(e instanceof CallSequenceOutOfOrderException)) {
{code}
Would it make sense testing ! DoNotRetryIOException rather than calling out the
two exceptions above? Would that be too broad? Otherwise, wondering if these
two exceptions should inherit from a common base class given they are getting
this special treatment. Not important. Just a thought.
Check your comments. You seem to be saying 'scan' when you mean 'next'.
For example, this comment:
{code}
+// increment the callSeq which will be getting used for the next
time scan() call to
+// the RS.In case of a timeout this increment should not happen so
that the next
+// trial also will be done with the same callSeq.
{code}
Regards the above comment, is there a sentence missing on what happens if we go
back w/ the same callseq? Maybe have another go at this whole comment... it
could be a bit clearer. Thanks.
Committing this set of comments more to come.
Scanner retry behavior with RPC timeout on next() seems incorrect
-
Key: HBASE-5974
URL: https://issues.apache.org/jira/browse/HBASE-5974
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
Reporter: Todd Lipcon
Assignee: Anoop Sam John
Priority: Critical
Fix For: 0.96.0
Attachments: 5974_94-V4.patch, 5974_trunk.patch, 5974_trunk-V2.patch,
HBASE-5974_0.94.patch, HBASE-5974_94-V2.patch, HBASE-5974_94-V3.patch
I'm seeing the following behavior:
- set RPC timeout to a short value
- call next() for some batch of rows, big enough so the client times out
before the result is returned
- the HConnectionManager stuff will retry the next() call to the same server.
At this point, one of two things can happen: 1) the previous next() call will
still be processing, in which case you get a LeaseException, because it was
removed from the map during the processing, or 2) the next() call will
succeed but skip the prior batch of rows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7009: Attachment: HBASE-7009-p1.patch For some reason git won't apply --no-prefix patch w/-p0 for me, but applies prefixed patch with -p1 just right. Port HBaseCluster interface/tests to 0.94 - Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7009-p1.patch, HBASE-7009.patch Need to port. I am porting V5 patch from the original JIRA; I have a partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479331#comment-13479331
]
Hadoop QA commented on HBASE-7009:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549747/HBASE-7009-p1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 70 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3081//console
This message is automatically generated.
Port HBaseCluster interface/tests to 0.94
-
Key: HBASE-7009
URL: https://issues.apache.org/jira/browse/HBASE-7009
Project: HBase
Issue Type: Sub-task
Components: test
Affects Versions: 0.94.3
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HBASE-7009-p1.patch, HBASE-7009.patch
Need to port. I am porting V5 patch from the original JIRA; I have a
partially ported (V3) patch from Enis with protocol buffers being reverted to
HRegionInterface/HMasterInterface
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7009) Port HBaseCluster interface/tests to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7009: Fix Version/s: 0.94.3 Port HBaseCluster interface/tests to 0.94 - Key: HBASE-7009 URL: https://issues.apache.org/jira/browse/HBASE-7009 Project: HBase Issue Type: Sub-task Components: test Affects Versions: 0.94.3 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.94.3 Attachments: HBASE-7009-p1.patch, HBASE-7009.patch Need to port. I am porting V5 patch from the original JIRA; I have a partially ported (V3) patch from Enis with protocol buffers being reverted to HRegionInterface/HMasterInterface -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6974: - Attachment: 6974-v5.txt Here's what I am going to commit soon. Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6974-v5.txt, HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5974) Scanner retry behavior with RPC timeout on next() seems incorrect
[
https://issues.apache.org/jira/browse/HBASE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479361#comment-13479361
]
stack commented on HBASE-5974:
--
Like Lars and Todd above I almost said why RegionScannerHolder? Is it because
if you add this class, you can do the seqid management in the regionserver and
CPs do not have to be concerned with its management? That is fair enough and
this is pretty elegant soln to the issue.
Your worry is that a CP might return its own RegionScanner implementation. Can
you not add to RegionScanner to methods, getNextSeq, and incrementNextSeq?
Then the regionserver would have the means for keeping up the seqid in whatever
the implementation rather than introduce this wrapper class? RegionScanner is
marked as private. For the trunk patch, we are requiring a restart and
allowing that CPs may need modification to make the leap from 0.94 to 0.96.
Does that make it easier on you modifying RegionScanner to support nextSeq?
Yeah, I think that rather than:
{code}
+ optional uint64 callSeq = 6;
{code}
it should be called nextSeq or nextCallSeq (the latter might be better because
nextSeq might make you think the next sequence id rather than the sequence id
for the 'next' call). All references should be changed if you change this Id
say Anoop.
This test, TestClientScannerRPCTimeout, looks good. Does it have to be in a
separate class? Do we not have a scanner test already that has a running
minicluster? I can understand though if you want to keep this separate because
its hard to integrate into a current test suite. If you are going to have a
class for this single test and are going to go to the bother of spinning up a
whole minicluster, I'd say do a bit more testing. Are there other scenarios
you can manufacture? Can you add assert that you actually slept?
Good stuff Anoop. Lets get this important patch in.
Scanner retry behavior with RPC timeout on next() seems incorrect
-
Key: HBASE-5974
URL: https://issues.apache.org/jira/browse/HBASE-5974
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Affects Versions: 0.90.7, 0.92.1, 0.94.0, 0.96.0
Reporter: Todd Lipcon
Assignee: Anoop Sam John
Priority: Critical
Fix For: 0.96.0
Attachments: 5974_94-V4.patch, 5974_trunk.patch, 5974_trunk-V2.patch,
HBASE-5974_0.94.patch, HBASE-5974_94-V2.patch, HBASE-5974_94-V3.patch
I'm seeing the following behavior:
- set RPC timeout to a short value
- call next() for some batch of rows, big enough so the client times out
before the result is returned
- the HConnectionManager stuff will retry the next() call to the same server.
At this point, one of two things can happen: 1) the previous next() call will
still be processing, in which case you get a LeaseException, because it was
removed from the map during the processing, or 2) the next() call will
succeed but skip the prior batch of rows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6577) RegionScannerImpl.nextRow() should seek to next row
[ https://issues.apache.org/jira/browse/HBASE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479372#comment-13479372 ] Jean-Daniel Cryans commented on HBASE-6577: --- Yeah I do remember that it's a thrift issue but we have 200 tables and 3 clusters that get reads through Thrift :( RegionScannerImpl.nextRow() should seek to next row --- Key: HBASE-6577 URL: https://issues.apache.org/jira/browse/HBASE-6577 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0 Attachments: 6577-0.94.txt, 6577.txt, 6577-v2.txt, 6577-v3.txt RegionScannerImpl.nextRow() is called when a filter filters the entire row. In that case we should seek to the next row rather then iterating over all versions of all columns to get there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4557) Do something better than UnknownScannerException
[
https://issues.apache.org/jira/browse/HBASE-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479375#comment-13479375
]
Ted Yu commented on HBASE-4557:
---
One a related note:
It is difficult for user to find out the correlation between region name and
scanner name when they see:
{code}
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: -3944363018500584369
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1869)
{code}
When entry keyed by scannerName is removed from scanners map, can we
temporarily store (scanner name, region name) mapping so that the above
exception message can provide more information to the user ?
Do something better than UnknownScannerException
Key: HBASE-4557
URL: https://issues.apache.org/jira/browse/HBASE-4557
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Fix For: 0.96.0
UnknownScannerException is a plague, there's no reason we should not at least
try to create a new scanner. If that fails again, maybe try automatically
setting a lower scanner caching (if possible and with proper loggin) and
retry again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kishore updated HBASE-6389:
--
Status: Open (was: Patch Available)
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kishore updated HBASE-6389:
--
Attachment: HBASE-6389_trunk_v2.patch
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kishore updated HBASE-6389:
--
Status: Patch Available (was: Open)
Resubmitting the patch with some minor changes.
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[
https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479409#comment-13479409
]
Hadoop QA commented on HBASE-6974:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549750/6974-v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3082//console
This message is automatically generated.
Metric for blocked updates
--
Key: HBASE-6974
URL: https://issues.apache.org/jira/browse/HBASE-6974
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Michael Drzal
Priority: Critical
Fix For: 0.94.3, 0.96.0
Attachments: 6974-v5.txt, HBASE-6974.patch, HBASE-6974-v2.patch,
HBASE-6974-v3.patch, HBASE-6974-v4.patch
When the disc subsystem cannot keep up with a sustained high write load, a
region will eventually block updates to throttle clients.
(HRegion.checkResources).
It would be nice to have a metric for this, so that these occurrences can be
tracked.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6974: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.94 and 0.96. Thanks for the patch, Drz. :) Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6974-v5.txt, HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Attachment: (was: ResourceChecker-0.94.patch) Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Status: Open (was: Patch Available) Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[ https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kumar Ravi updated HBASE-6945: -- Status: Patch Available (was: Open) Compilation errors when using non-Sun JDKs to build HBase-0.94 -- Key: HBASE-6945 URL: https://issues.apache.org/jira/browse/HBASE-6945 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.94.1 Environment: RHEL 6.3, IBM Java 7 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3 Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch When using IBM Java 7 to build HBase-0.94.1, the following comilation error is seen. [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25] error: package com.sun.management does not exist [ERROR] /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29] error: cannot find symbol [ERROR] symbol: class UnixOperatingSystemMXBean location: class ResourceAnalyzer /home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23] error: cannot find symbol [INFO] 4 errors [INFO] - [INFO] [INFO] BUILD FAILURE [INFO] I have a patch available which should work for all JDKs including Sun. I am in the process of testing this patch. Preliminary tests indicate the build is working fine with this patch. I will post this patch when I am done testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6758: - Attachment: 6758-0.94.txt 0.94 patch. The trunk patch applied with some fuzz, and FSHlog does not exist in 0.94. [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6758-0.94.txt, 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6758: - Fix Version/s: 0.94.3 [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6758-0.94.txt, 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7011) log rolling and cache flushing should be able to proceed in parallel
Kannan Muthukkaruppan created HBASE-7011: Summary: log rolling and cache flushing should be able to proceed in parallel Key: HBASE-7011 URL: https://issues.apache.org/jira/browse/HBASE-7011 Project: HBase Issue Type: Improvement Reporter: Kannan Muthukkaruppan Today, during a memstore flush (snapshot of memstore + flushing to disk), log rolling cannot happen. This seems like a bad design, and an unnecessary restriction. Possible reasons cited for this in code are: (i) maintenance of the lastSeqWritten map. (ii) writing a completed-cache-flush marker into the same log before the roll. It seems that we can implement a new design for (i) to avoid holding the lock for the entire duration of the flush. And the motivation for (ii) is not even clear. We should reason this out, and make sure we can relax the restriction. [See related discussion in HBASE-6980.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7011) log rolling and cache flushing should be able to proceed in parallel
[ https://issues.apache.org/jira/browse/HBASE-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kannan Muthukkaruppan updated HBASE-7011: - Assignee: Kannan Muthukkaruppan log rolling and cache flushing should be able to proceed in parallel Key: HBASE-7011 URL: https://issues.apache.org/jira/browse/HBASE-7011 Project: HBase Issue Type: Improvement Reporter: Kannan Muthukkaruppan Assignee: Kannan Muthukkaruppan Today, during a memstore flush (snapshot of memstore + flushing to disk), log rolling cannot happen. This seems like a bad design, and an unnecessary restriction. Possible reasons cited for this in code are: (i) maintenance of the lastSeqWritten map. (ii) writing a completed-cache-flush marker into the same log before the roll. It seems that we can implement a new design for (i) to avoid holding the lock for the entire duration of the flush. And the motivation for (ii) is not even clear. We should reason this out, and make sure we can relax the restriction. [See related discussion in HBASE-6980.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6962) Upgrade hadoop 1 dependency to hadoop 1.1
[ https://issues.apache.org/jira/browse/HBASE-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479433#comment-13479433 ] Lars Hofhansl commented on HBASE-6962: -- Should we update the default for 0.94 to 1.0.4 (and add a target for 1.1.0)? Upgrade hadoop 1 dependency to hadoop 1.1 - Key: HBASE-6962 URL: https://issues.apache.org/jira/browse/HBASE-6962 Project: HBase Issue Type: Bug Environment: hadoop 1.1 contains multiple important fixes, including HDFS-3703 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6962.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6965) Generic MXBean Utility class to support all JDK vendors
[ https://issues.apache.org/jira/browse/HBASE-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479441#comment-13479441 ] stack commented on HBASE-6965: -- Here are some comments on this patch (Thanks for doing it). In future, would suggest putting the two patches together... Having them separate makes it hard to review; I have to flip between two JIRAs to figure whether this thing of use or not? The class is misnamed. This is not a management bean. What JVMs and OS's did you test on out of interest? How many different vendor and OS strings did you test your patch against? It seems a big hacky looking for 'IBM' in vendor string figuring if an IBM JVM or not. Are you sure it's always upper case. Any other property you could check to be sure it is the JVM you think. Does IBM only make a linux JDK? Generic MXBean Utility class to support all JDK vendors --- Key: HBASE-6965 URL: https://issues.apache.org/jira/browse/HBASE-6965 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.1 Reporter: Kumar Ravi Assignee: Kumar Ravi Labels: patch Fix For: 0.94.3, 0.96.0 Attachments: HBASE-6965.patch, OSMXBean_HBASE-6965-0.94.patch This issue is related to JIRA https://issues.apache.org/jira/browse/HBASE-6945. This issue is opened to propose the use of a newly created generic org.apache.hadoop.hbase.util.OSMXBean class that can be used by other classes. JIRA HBASE-6945 contains a patch for the class org.apache.hadoop.hbase.ResourceChecker that uses OSMXBean. With the inclusion of this new class, HBase can be built and become functional with JDKs and JREs other than what is provided by Oracle. This class uses reflection to determine the JVM vendor (Sun, IBM) and the platform (Linux or Windows), and contains other methods that return the OS properties - 1. Number of Open File descriptors; 2. Maximum number of File Descriptors. This class compiles without any problems with IBM JDK 7, OpenJDK 6 as well as Oracle JDK 6. Junit tests (runDevTests category) completed without any failures or errors when tested on all the three JDKs.The builds and tests were attempted on branch hbase-0.94 Revision 1396305. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479443#comment-13479443
]
Hadoop QA commented on HBASE-6389:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549763/HBASE-6389_trunk_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3083//console
This message is automatically generated.
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will
[jira] [Created] (HBASE-7012) Create hbase-client module
Elliott Clark created HBASE-7012: Summary: Create hbase-client module Key: HBASE-7012 URL: https://issues.apache.org/jira/browse/HBASE-7012 Project: HBase Issue Type: Task Reporter: Elliott Clark I just tried creating a project that uses 0.95-SNAPSHOT and had to import org.apache.hbase:hbase-server as the module that I depend on. This will be confusing to users. In addition this brings in lots of dependencies that are not really needed. Let's create a client module that has all of the client in it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6962) Upgrade hadoop 1 dependency to hadoop 1.1
[ https://issues.apache.org/jira/browse/HBASE-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479448#comment-13479448 ] stack commented on HBASE-6962: -- @Enis You say, 'Although that Hadoop section needs a cleanup in general.' What would you suggest? @Lars We could upgrade yes. Anyone know if API differences between hadoop 1.1 and 1.0? Would it be better to ship w/ hadoop 1.0 since that is what we are saying is the minimum requirement for 0.96 and then recommend in doc. that folks run 1.1 because of the fixed issues and perf benefits? As has been suggested above, seems like this is something worthy of raising on dev list (even if this has gone in already, if only to raise consciousness). Upgrade hadoop 1 dependency to hadoop 1.1 - Key: HBASE-6962 URL: https://issues.apache.org/jira/browse/HBASE-6962 Project: HBase Issue Type: Bug Environment: hadoop 1.1 contains multiple important fixes, including HDFS-3703 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6962.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479450#comment-13479450 ] Lars Hofhansl commented on HBASE-6758: -- any objections/concerns with committing this to 0.94? [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6758-0.94.txt, 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479451#comment-13479451
]
Aditya Kishore commented on HBASE-6389:
---
The test failure is unrelated and most likely is due to the same reason as
observed here
https://issues.apache.org/jira/browse/HBASE-6707?focusedCommentId=13469675page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13469665
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[
https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479460#comment-13479460
]
Hadoop QA commented on HBASE-6945:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12549710/ResourceCheckerJUnitListener_HBASE_6945-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
82 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3084//console
This message is automatically generated.
Compilation errors when using non-Sun JDKs to build HBase-0.94
--
Key: HBASE-6945
URL: https://issues.apache.org/jira/browse/HBASE-6945
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.94.1
Environment: RHEL 6.3, IBM Java 7
Reporter: Kumar Ravi
Assignee: Kumar Ravi
Labels: patch
Fix For: 0.94.3
Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch
When using IBM Java 7 to build HBase-0.94.1, the following comilation error
is seen.
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25]
error: package com.sun.management does not exist
[ERROR]
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25]
error: cannot find symbol
[ERROR] symbol: class UnixOperatingSystemMXBean
location: class ResourceAnalyzer
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29]
error: cannot find symbol
[ERROR] symbol: class UnixOperatingSystemMXBean
location: class ResourceAnalyzer
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23]
error: cannot find symbol
[INFO] 4 errors
[INFO] -
[INFO]
[INFO] BUILD FAILURE
[INFO]
I have a patch available which should work for all JDKs including Sun.
I am in the process of testing this patch. Preliminary tests indicate the
build is working fine with this patch. I will post this patch when I am done
testing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479462#comment-13479462 ] Hudson commented on HBASE-6974: --- Integrated in HBase-TRUNK #3459 (See [https://builds.apache.org/job/HBase-TRUNK/3459/]) HBASE-6974 Metric for blocked updates (Michael Drzal) (Revision 1399880) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6974-v5.txt, HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6389) Modify the conditions to ensure that Master waits for sufficient number of Region Servers before starting region assignments
[
https://issues.apache.org/jira/browse/HBASE-6389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479464#comment-13479464
]
stack commented on HBASE-6389:
--
[~adityakishore] Patch looks good Aditya. Mind describing in release note how
this patch changes startup? What kind of testing do you do sir? I can commit
to trunk. Should it go into 0.94 too?
Modify the conditions to ensure that Master waits for sufficient number of
Region Servers before starting region assignments
Key: HBASE-6389
URL: https://issues.apache.org/jira/browse/HBASE-6389
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.0, 0.96.0
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Priority: Critical
Fix For: 0.96.0
Attachments: HBASE-6389_trunk.patch, HBASE-6389_trunk.patch,
HBASE-6389_trunk.patch, HBASE-6389_trunk_v2.patch, HBASE-6389_trunk_v2.patch,
org.apache.hadoop.hbase.TestZooKeeper-output.txt, testReplication.jstack
Continuing from HBASE-6375.
It seems I was mistaken in my assumption that changing the value of
hbase.master.wait.on.regionservers.mintostart to a sufficient number (from
default of 1) can help prevent assignment of all regions to one (or a small
number of) region server(s).
While this was the case in 0.90.x and 0.92.x, the behavior has changed in
0.94.0 onwards to address HBASE-4993.
From 0.94.0 onwards, Master will proceed immediately after the timeout has
lapsed, even if hbase.master.wait.on.regionservers.mintostart has not
reached.
Reading the current conditions of waitForRegionServers() clarifies it
{code:title=ServerManager.java (trunk rev:1360470)}
581 /**
582 * Wait for the region servers to report in.
583 * We will wait until one of this condition is met:
584 * - the master is stopped
585 * - the 'hbase.master.wait.on.regionservers.timeout' is reached
586 * - the 'hbase.master.wait.on.regionservers.maxtostart' number of
587 *region servers is reached
588 * - the 'hbase.master.wait.on.regionservers.mintostart' is reached
AND
589 * there have been no new region server in for
590 * 'hbase.master.wait.on.regionservers.interval' time
591 *
592 * @throws InterruptedException
593 */
594 public void waitForRegionServers(MonitoredTask status)
595 throws InterruptedException {
612 while (
613 !this.master.isStopped()
614 slept timeout
615 count maxToStart
616 (lastCountChange+interval now || count minToStart)
617 ){
{code}
So with the current conditions, the wait will end as soon as timeout is
reached even lesser number of RS have checked-in with the Master and the
master will proceed with the region assignment among these RSes alone.
As mentioned in
-[HBASE-4993|https://issues.apache.org/jira/browse/HBASE-4993?focusedCommentId=13237196#comment-13237196]-,
and I concur, this could have disastrous effect in large cluster especially
now that MSLAB is turned on.
To enforce the required quorum as specified by
hbase.master.wait.on.regionservers.mintostart irrespective of timeout,
these conditions need to be modified as following
{code:title=ServerManager.java}
..
/**
* Wait for the region servers to report in.
* We will wait until one of this condition is met:
* - the master is stopped
* - the 'hbase.master.wait.on.regionservers.maxtostart' number of
*region servers is reached
* - the 'hbase.master.wait.on.regionservers.mintostart' is reached AND
* there have been no new region server in for
* 'hbase.master.wait.on.regionservers.interval' time AND
* the 'hbase.master.wait.on.regionservers.timeout' is reached
*
* @throws InterruptedException
*/
public void waitForRegionServers(MonitoredTask status)
..
..
int minToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.mintostart, 1);
int maxToStart = this.master.getConfiguration().
getInt(hbase.master.wait.on.regionservers.maxtostart,
Integer.MAX_VALUE);
if (maxToStart minToStart) {
maxToStart = minToStart;
}
..
..
while (
!this.master.isStopped()
count maxToStart
(lastCountChange+interval now || timeout slept || count
minToStart)
){
..
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6974) Metric for blocked updates
[ https://issues.apache.org/jira/browse/HBASE-6974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479465#comment-13479465 ] Hudson commented on HBASE-6974: --- Integrated in HBase-0.94 #541 (See [https://builds.apache.org/job/HBase-0.94/541/]) HBASE-6974 Metric for blocked updates (Michael Drzal) (Revision 1399881) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java Metric for blocked updates -- Key: HBASE-6974 URL: https://issues.apache.org/jira/browse/HBASE-6974 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Michael Drzal Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6974-v5.txt, HBASE-6974.patch, HBASE-6974-v2.patch, HBASE-6974-v3.patch, HBASE-6974-v4.patch When the disc subsystem cannot keep up with a sustained high write load, a region will eventually block updates to throttle clients. (HRegion.checkResources). It would be nice to have a metric for this, so that these occurrences can be tracked. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6945) Compilation errors when using non-Sun JDKs to build HBase-0.94
[
https://issues.apache.org/jira/browse/HBASE-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479469#comment-13479469
]
stack commented on HBASE-6945:
--
Does this patch make a class with no content? I'm looking at this section:
{code}
abstract static class OSResourceAnalyzer extends
ResourceChecker.ResourceAnalyzer {
-protected static final OperatingSystemMXBean osStats;
-protected static final UnixOperatingSystemMXBean unixOsStats;
-
-static {
- osStats = ManagementFactory.getOperatingSystemMXBean();
- if (osStats instanceof UnixOperatingSystemMXBean) {
-unixOsStats = (UnixOperatingSystemMXBean) osStats;
- } else {
-unixOsStats = null;
- }
-}
}
{code}
We seem to be removing the body of the class.
Should this class, OSMXBean, be renamed OS since it answers questions about the
OS in a way that insulates us against differences in JVM. Maybe a better name
would be JVM. Then you'd ask it for an implementation of
UnixOperatingSystemMXBean. It would take care of returning the IBM or Oracle
implementation. They both implement the UnixOperatingSystemMXBean Interface?
Compilation errors when using non-Sun JDKs to build HBase-0.94
--
Key: HBASE-6945
URL: https://issues.apache.org/jira/browse/HBASE-6945
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.94.1
Environment: RHEL 6.3, IBM Java 7
Reporter: Kumar Ravi
Assignee: Kumar Ravi
Labels: patch
Fix For: 0.94.3
Attachments: ResourceCheckerJUnitListener_HBASE_6945-trunk.patch
When using IBM Java 7 to build HBase-0.94.1, the following comilation error
is seen.
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[23,25]
error: package com.sun.management does not exist
[ERROR]
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[46,25]
error: cannot find symbol
[ERROR] symbol: class UnixOperatingSystemMXBean
location: class ResourceAnalyzer
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[75,29]
error: cannot find symbol
[ERROR] symbol: class UnixOperatingSystemMXBean
location: class ResourceAnalyzer
/home/hadoop/hbase-0.94/src/test/java/org/apache/hadoop/hbase/ResourceChecker.java:[76,23]
error: cannot find symbol
[INFO] 4 errors
[INFO] -
[INFO]
[INFO] BUILD FAILURE
[INFO]
I have a patch available which should work for all JDKs including Sun.
I am in the process of testing this patch. Preliminary tests indicate the
build is working fine with this patch. I will post this patch when I am done
testing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479471#comment-13479471 ] Jean-Daniel Cryans commented on HBASE-6758: --- +1 if it's tested on a cluster. [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6758-0.94.txt, 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13479474#comment-13479474 ] stack commented on HBASE-6758: -- Is that a non-change in Index: src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java? Good by me committing to 0.94. [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.94.3, 0.96.0 Attachments: 6758-0.94.txt, 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, 6758-trunk-2.patch, 6758-trunk-3.patch, 6758-trunk-4.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
