[jira] [Commented] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596222#comment-14596222
]
Anoop Sam John commented on HBASE-13945:
Also the test case in TestDataBlockEncoder need change. assertEquals is not
asserting the bytes from actualKey and expectedKey. The BB's position will be
at the end and so not checking the actual key bytes. In tests we should rewind
both these BBs and then do assertEquals
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.2, 1.0.1, 1.1.0, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_0.98_2.patch, HBASE-13945_branch-1.1.patch,
HBASE-13945_trunk.patch, HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13017) Backport HBASE-12035 Keep table state in Meta to branch-1
[ https://issues.apache.org/jira/browse/HBASE-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596234#comment-14596234 ] Sean Busbey commented on HBASE-13017: - thanks for the update. I added fix versions to HBASE-13147. If we can't get it to work in a backwards compatible way we can update the fix version and close this one out. Backport HBASE-12035 Keep table state in Meta to branch-1 - Key: HBASE-13017 URL: https://issues.apache.org/jira/browse/HBASE-13017 Project: HBase Issue Type: Improvement Components: master Affects Versions: 1.1.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Labels: backport Fix For: 1.3.0 Attachments: HBASE-13017-branch-1.patch, HBASE-13017-branch-1.v1.patch, HBASE-13017-branch-1.v1.patch, HBASE-13017-branch-1.v2.patch, HBASE-13017-branch-1.v3.patch, HBASE-13017-branch-1.v4.patch, HBASE-13017-branch-1.v5.patch, HBASE-13017-branch-1.v6.patch Lets backport that feature to branch-1.0 adapting HBASE-12035 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12636) Avoid too many write operations on zookeeper in replication
[ https://issues.apache.org/jira/browse/HBASE-12636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12636: Fix Version/s: (was: 1.2.0) Current patch appears to address specific feedback points to date and is an incremental improvement. I'm +1 for master unless someone has further objections. [~liushaohui] can you get me a rebased patch for master? the current one doesn't apply. Avoid too many write operations on zookeeper in replication --- Key: HBASE-12636 URL: https://issues.apache.org/jira/browse/HBASE-12636 Project: HBase Issue Type: Improvement Affects Versions: 0.94.11 Reporter: Liu Shaohui Assignee: Liu Shaohui Labels: replication Fix For: 2.0.0 Attachments: HBASE-12635-v2.diff, HBASE-12636-v1.diff In our production cluster, we found there are about over 1k write operations per second on zookeeper from hbase replication. The reason is that the replication source will write the log position to zookeeper for every edit shipping. If the current replicating WAL is just the WAL that regionserver is writing to, each skipping will be very small but the frequency is very high, which causes many write operations on zookeeper. A simple solution is that writing log position to zookeeper when position diff or skipped edit number is larger than a threshold, not every edit shipping. Suggestions are welcomed, thx~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10070) HBase read high-availability using timeline-consistent region replicas
[ https://issues.apache.org/jira/browse/HBASE-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596299#comment-14596299 ] Sean Busbey commented on HBASE-10070: - what's current status on this, specifically in branch-1 for the 1.2 release? HBase read high-availability using timeline-consistent region replicas -- Key: HBASE-10070 URL: https://issues.apache.org/jira/browse/HBASE-10070 Project: HBase Issue Type: New Feature Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 2.0.0, 1.2.0 Attachments: HighAvailabilityDesignforreadsApachedoc.pdf In the present HBase architecture, it is hard, probably impossible, to satisfy constraints like 99th percentile of the reads will be served under 10 ms. One of the major factors that affects this is the MTTR for regions. There are three phases in the MTTR process - detection, assignment, and recovery. Of these, the detection is usually the longest and is presently in the order of 20-30 seconds. During this time, the clients would not be able to read the region data. However, some clients will be better served if regions will be available for reads during recovery for doing eventually consistent reads. This will help with satisfying low latency guarantees for some class of applications which can work with stale reads. For improving read availability, we propose a replicated read-only region serving design, also referred as secondary regions, or region shadows. Extending current model of a region being opened for reads and writes in a single region server, the region will be also opened for reading in region servers. The region server which hosts the region for reads and writes (as in current case) will be declared as PRIMARY, while 0 or more region servers might be hosting the region as SECONDARY. There may be more than one secondary (replica count 2). Will attach a design doc shortly which contains most of the details and some thoughts about development approaches. Reviews are more than welcome. We also have a proof of concept patch, which includes the master and regions server side of changes. Client side changes will be coming soon as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13900) duplicate methods between ProtobufMagic and ProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-13900. Resolution: Fixed [~misty] did the necessary revert an recommit already. Thanks [~misty]! duplicate methods between ProtobufMagic and ProtobufUtil Key: HBASE-13900 URL: https://issues.apache.org/jira/browse/HBASE-13900 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Assignee: Gabor Liptak Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13900.1.patch Several methods duplicated between: hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufMagic.java and hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java isPBMagicPrefix() isPBMagicPrefix() lengthOfPBMagic() ProtobufUtil partically references ProtobufMagic, but it also has different compare implementation ... Maybe this was based on packaging/signature need? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596217#comment-14596217
]
Anoop Sam John commented on HBASE-13945:
Should have checked this..
{quote}
/** @return key value at current position with position set to limit */
ByteBuffer getKeyValueBuffer();
{quote}
Other DBEs return BB with pos at end. That is why Prefix tree also implemented
this way.
bq. I was thinking whether we should leave the KVUtil method as is and at used
place do a BB#rewind()
Seems this is the correct way.
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.2, 1.0.1, 1.1.0, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_0.98_2.patch, HBASE-13945_branch-1.1.patch,
HBASE-13945_trunk.patch, HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596272#comment-14596272
]
Hadoop QA commented on HBASE-13945:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12741045/HBASE-13945_0.98_2.patch
against 0.98 branch at commit d51a184051d968dc3bdc00b1c9257c0a9e5ff8a6.
ATTACHMENT ID: 12741045
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
24 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14502//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14502//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14502//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14502//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14502//console
This message is automatically generated.
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.2, 1.0.1, 1.1.0, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_0.98_2.patch, HBASE-13945_branch-1.1.patch,
HBASE-13945_trunk.patch, HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13925) Use zookeeper multi to clear znodes in ZKProcedureUtil
[ https://issues.apache.org/jira/browse/HBASE-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13925: Fix Version/s: (was: 1.2.0) Use zookeeper multi to clear znodes in ZKProcedureUtil -- Key: HBASE-13925 URL: https://issues.apache.org/jira/browse/HBASE-13925 Project: HBase Issue Type: Improvement Affects Versions: 0.98.13 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-13925-v1-again.patch, HBASE-13925-v1.patch, HBASE-13925.patch Address the TODO in ZKProcedureUtil clearChildZNodes() and clearZNodes methods -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13158: Fix Version/s: (was: 1.2.0) 1.3.0 When client supports CellBlock, return the result Cells as controller payload for get(Get) API also --- Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 1.3.0 Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596191#comment-14596191
]
Andrew Purtell commented on HBASE-13945:
bq. The test case failure was due to the change in API whcih was getting used
in the tests. Not a functional issue. Should be good to commit.
Pardon, what do you mean Ram? Let's not commit something that will cause unit
tests to fail. Thanks.
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.2, 1.0.1, 1.1.0, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_0.98_2.patch, HBASE-13945_branch-1.1.patch,
HBASE-13945_trunk.patch, HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13017) Backport HBASE-12035 Keep table state in Meta to branch-1
[ https://issues.apache.org/jira/browse/HBASE-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596215#comment-14596215 ] Andrey Stepachev commented on HBASE-13017: -- [~busbey] Not sure is it possible to make that in branch-1. I was stuck with HBASE-13147 which is needed to distinguish old metas from new metas. If we'll find a way to make that backward compatible (as in HBASE-13147) for branch-1, so we can apply this patch once more. Backport HBASE-12035 Keep table state in Meta to branch-1 - Key: HBASE-13017 URL: https://issues.apache.org/jira/browse/HBASE-13017 Project: HBase Issue Type: Improvement Components: master Affects Versions: 1.1.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Labels: backport Fix For: 1.3.0 Attachments: HBASE-13017-branch-1.patch, HBASE-13017-branch-1.v1.patch, HBASE-13017-branch-1.v1.patch, HBASE-13017-branch-1.v2.patch, HBASE-13017-branch-1.v3.patch, HBASE-13017-branch-1.v4.patch, HBASE-13017-branch-1.v5.patch, HBASE-13017-branch-1.v6.patch Lets backport that feature to branch-1.0 adapting HBASE-12035 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13832) Procedure V2: master fail to start due to WALProcedureStore sync failures when HDFS data nodes count is low
[
https://issues.apache.org/jira/browse/HBASE-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596245#comment-14596245
]
Ted Yu commented on HBASE-13832:
bq. hbase.procedure.store.wal.max.retry.before.roll;
'retry' - 'retries'
{code}
611 } catch (IOException e) {
612 LOG.warn(Unable to roll the log, e);
{code}
Please log i, the roll retry count.
{code}
147 public static class TestProcedure extends ProcedureVoid {
{code}
This class can be private, right ?
Procedure V2: master fail to start due to WALProcedureStore sync failures
when HDFS data nodes count is low
---
Key: HBASE-13832
URL: https://issues.apache.org/jira/browse/HBASE-13832
Project: HBase
Issue Type: Sub-task
Components: master, proc-v2
Affects Versions: 2.0.0, 1.1.0, 1.2.0
Reporter: Stephen Yuan Jiang
Assignee: Matteo Bertozzi
Attachments: HBASE-13832-v0.patch, HDFSPipeline.java
when the data node 3, we got failure in WALProcedureStore#syncLoop() during
master start. The failure prevents master to get started.
{noformat}
2015-05-29 13:27:16,625 ERROR [WALProcedureStoreSyncThread]
wal.WALProcedureStore: Sync slot failed, abort.
java.io.IOException: Failed to replace a bad datanode on the existing
pipeline due to no more good datanodes being available to try. (Nodes:
current=[DatanodeInfoWithStorage[10.333.444.555:50010,DS-3ced-93f4-47b6-9c23-1426f7a6acdc,DISK],
DatanodeInfoWithStorage[10.222.666.777:50010,DS-f9c983b4-1f10-4d5e-8983-490ece56c772,DISK]],
original=[DatanodeInfoWithStorage[10.333.444.555:50010,DS-3ced-93f4-47b6-9c23-1426f7a6acdc,DISK],
DatanodeInfoWithStorage[10.222.666.777:50010,DS-f9c983b4-1f10-4d5e-8983-
490ece56c772,DISK]]). The current failed datanode replacement policy is
DEFAULT, and a client may configure this via
'dfs.client.block.write.replace-datanode-on-failure.policy' in its
configuration.
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:951)
{noformat}
One proposal is to implement some similar logic as FSHLog: if IOException is
thrown during syncLoop in WALProcedureStore#start(), instead of immediate
abort, we could try to roll the log and see whether this resolve the issue;
if the new log cannot be created or more exception from rolling the log, we
then abort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13889) hbase-shaded-client artifact is missing dependency (therefore, does not work)
[
https://issues.apache.org/jira/browse/HBASE-13889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596163#comment-14596163
]
Nick Dimiduk commented on HBASE-13889:
--
I haven't spent any more time on this since my last update. Won't be in for
1.1.1, but I'll maybe have a go at debugging the {{com.sun}} stuff after RC.
hbase-shaded-client artifact is missing dependency (therefore, does not work)
-
Key: HBASE-13889
URL: https://issues.apache.org/jira/browse/HBASE-13889
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 1.1.0, 1.1.0.1
Environment: N/A?
Reporter: Dmitry Minkovsky
Priority: Blocker
Fix For: 2.0.0, 1.2.0, 1.1.2
Attachments: 13889.wip.patch, Screen Shot 2015-06-11 at 10.59.55
AM.png
The {{hbase-shaded-client}} artifact was introduced in
[HBASE-13517|https://issues.apache.org/jira/browse/HBASE-13517]. Thank you
very much for this, as I am new to Java building and was having a very
slow-moving time resolving conflicts. However, the shaded client artifact
seems to be missing {{javax.xml.transform.TransformerException}}. I examined
the JAR, which does not have this package/class.
Steps to reproduce:
Java:
{code}
package com.mycompany.app;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
public class App {
public static void main( String[] args ) throws java.io.IOException {
Configuration config = HBaseConfiguration.create();
Connection connection = ConnectionFactory.createConnection(config);
}
}
{code}
POM:
{code}
project xmlns=http://maven.apache.org/POM/4.0.0;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
modelVersion4.0.0/modelVersion
groupIdcom.mycompany.app/groupId
artifactIdmy-app/artifactId
version1.0-SNAPSHOT/version
[jira] [Updated] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13259: Fix Version/s: (was: 1.2.0) 1.3.0 bumping out to 1.3 mmap() based BucketCache IOEngine - Key: HBASE-13259 URL: https://issues.apache.org/jira/browse/HBASE-13259 Project: HBase Issue Type: New Feature Components: BlockCache Affects Versions: 0.98.10 Reporter: Zee Chen Fix For: 2.0.0, 1.3.0 Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. This patch modifies ByteByfferArray to allow it to specify a backing file. Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13336) Consistent rules for security meta table protections
[ https://issues.apache.org/jira/browse/HBASE-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13336: Fix Version/s: (was: 1.2.0) 1.3.0 Consistent rules for security meta table protections Key: HBASE-13336 URL: https://issues.apache.org/jira/browse/HBASE-13336 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Srikanth Srungarapu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-13336.patch The AccessController and VisibilityController do different things regarding protecting their meta tables. The AC allows schema changes and disable/enable if the user has permission. The VC unconditionally disallows all admin actions. Generally, bad things will happen if these meta tables are damaged, disabled, or dropped. The likely outcome is random frequent (or constant) server side op failures with nasty stack traces. On the other hand some things like column family and table attribute changes can have valid use cases. We should have consistent and sensible rules for protecting security meta tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13031) Ability to snapshot based on a key range
[ https://issues.apache.org/jira/browse/HBASE-13031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13031: Fix Version/s: (was: 1.2.0) 1.3.0 Ability to snapshot based on a key range Key: HBASE-13031 URL: https://issues.apache.org/jira/browse/HBASE-13031 Project: HBase Issue Type: Improvement Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 0.94.26, 0.98.14, 1.3.0 Attachments: HBASE-13031-v1.patch, HBASE-13031.patch Posted on the mailing list and seems like some people are interested. A little background for everyone. We have a very large table, we would like to snapshot and transfer the data to another cluster (compressed data is always better to ship). Our problem lies in the fact it could take many weeks to transfer all of the data and during that time with major compactions, the data stored in dfs has the potential to double which would cause us to run out of disk space. So we were thinking about allowing the ability to snapshot a specific key range. Ideally I feel the approach is that the user would specify a start and stop key, those would be associated with a region boundary. If between the time the user submits the request and the snapshot is taken the boundaries change (due to merging or splitting of regions) the snapshot should fail. We would know which regions to snapshot and if those changed between when the request was submitted and the regions locked, the snapshot could simply fail and the user would try again, instead of potentially giving the user more / less than what they had anticipated. I was planning on storing the start / stop key in the SnapshotDescription and from there it looks pretty straight forward where we just have to change the verifier code to accommodate the key ranges. If this design sounds good to anyone, or if I am overlooking anything please let me know. Once we agree on the design, I'll write and submit the patches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13014) Java Tool For Region Moving
[ https://issues.apache.org/jira/browse/HBASE-13014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596204#comment-14596204 ] Sean Busbey commented on HBASE-13014: - is this good to go? [~lhofhansl], [~apurtell], [~haridsv]? Java Tool For Region Moving Key: HBASE-13014 URL: https://issues.apache.org/jira/browse/HBASE-13014 Project: HBase Issue Type: Improvement Reporter: Abhishek Singh Chouhan Assignee: Abhishek Singh Chouhan Fix For: 2.0.0, 0.98.14, 1.2.0 Attachments: HBASE-13014-v2.patch, HBASE-13014-v3.patch, HBASE-13014-v4.patch, HBASE-13014-v5.patch, HBASE-13014-v6.patch, HBASE-13014.patch As per discussion on HBASE-12989 we should move the functionality of region_mover.rb into a Java tool and use region_mover.rb only only as a wrapper around it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13622) document upgrade rollback
[ https://issues.apache.org/jira/browse/HBASE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13622: Status: Patch Available (was: Open) document upgrade rollback - Key: HBASE-13622 URL: https://issues.apache.org/jira/browse/HBASE-13622 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 1.1.0, 1.0.0, 0.98.0 Reporter: Sean Busbey Assignee: Sean Busbey Labels: operability, upgrade Fix For: 1.2.0 Attachments: HBASE-13622.1.patch I have some docs on doing a rollback of an hbase upgrade that are currently vendor specific. polish them up, make them suitable for HBase generally, and add them to the ref guide. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13622) document upgrade rollback
[ https://issues.apache.org/jira/browse/HBASE-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13622: Attachment: HBASE-13622.1.patch built site locally and read through things. document upgrade rollback - Key: HBASE-13622 URL: https://issues.apache.org/jira/browse/HBASE-13622 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.98.0, 1.0.0, 1.1.0 Reporter: Sean Busbey Assignee: Sean Busbey Labels: operability, upgrade Fix For: 1.2.0 Attachments: HBASE-13622.1.patch I have some docs on doing a rollback of an hbase upgrade that are currently vendor specific. polish them up, make them suitable for HBase generally, and add them to the ref guide. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596289#comment-14596289
]
ramkrishna.s.vasudevan commented on HBASE-13945:
bq. Let's not commit something that will cause unit tests to fail. Thanks.
The unit test failed because of a change in the API that was used only in test
case. So there is no functional issue.
bq.ByteBuffer getKeyValueBuffer();
True, but no where the usage is as per the doc. As you said the
TestDataBlockEncoders was already not asserting the BB fully which always
needed a rewind as per the BB's equal impl. That is a valid point. We can
change that.
This getKeyValueBuffer() is used no where except in tests so better to remove
it or fix the API doc.
Also the KVUtil API functionality wise copying the KV to a BB and making
position == limit is making the BB non functional. (It is making it a something
like an EMPTY byte[]).
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.2, 1.0.1, 1.1.0, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_0.98_2.patch, HBASE-13945_branch-1.1.patch,
HBASE-13945_trunk.patch, HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13889) hbase-shaded-client artifact is missing dependency (therefore, does not work)
[
https://issues.apache.org/jira/browse/HBASE-13889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-13889:
Fix Version/s: (was: 1.2.0)
1.2.1
1.3.0
hbase-shaded-client artifact is missing dependency (therefore, does not work)
-
Key: HBASE-13889
URL: https://issues.apache.org/jira/browse/HBASE-13889
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 1.1.0, 1.1.0.1
Environment: N/A?
Reporter: Dmitry Minkovsky
Priority: Blocker
Fix For: 2.0.0, 1.1.2, 1.3.0, 1.2.1
Attachments: 13889.wip.patch, Screen Shot 2015-06-11 at 10.59.55
AM.png
The {{hbase-shaded-client}} artifact was introduced in
[HBASE-13517|https://issues.apache.org/jira/browse/HBASE-13517]. Thank you
very much for this, as I am new to Java building and was having a very
slow-moving time resolving conflicts. However, the shaded client artifact
seems to be missing {{javax.xml.transform.TransformerException}}. I examined
the JAR, which does not have this package/class.
Steps to reproduce:
Java:
{code}
package com.mycompany.app;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Connection;
import org.apache.hadoop.hbase.client.ConnectionFactory;
public class App {
public static void main( String[] args ) throws java.io.IOException {
Configuration config = HBaseConfiguration.create();
Connection connection = ConnectionFactory.createConnection(config);
}
}
{code}
POM:
{code}
project xmlns=http://maven.apache.org/POM/4.0.0;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation=http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;
modelVersion4.0.0/modelVersion
groupIdcom.mycompany.app/groupId
artifactIdmy-app/artifactId
version1.0-SNAPSHOT/version
packagingjar/packaging
[jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596243#comment-14596243
]
Andrew Purtell commented on HBASE-13935:
Thanks for the patch!
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12986) Compaction pressure based client pushback
[
https://issues.apache.org/jira/browse/HBASE-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-12986:
Fix Version/s: (was: 1.2.0)
1.3.0
Compaction pressure based client pushback
-
Key: HBASE-12986
URL: https://issues.apache.org/jira/browse/HBASE-12986
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 1.3.0
HBASE-8329 recently introduced on all branches {{double
RegionServerServices#getCompactionPressure()}}, which returns a value greater
than or equal to 0.0, and any value greater than 1.0 means we have exceeded
the store file limit on some stores. It could be reasonable to send this
value along in server load statistics (clamping max at 1.0), and consider it
as an additional term in the ExponentialClientBackoffPolicy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596276#comment-14596276
]
Nick Dimiduk commented on HBASE-13938:
--
Patch applies cleanly to branch-1 and branch-1.2 (FYI [~busbey]). Resolving
conflicts for master.
[~devaraj], [~enis] this needed for branch-1.0 and 0.98 also?
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596145#comment-14596145 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596143#comment-14596143 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596140#comment-14596140 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596139#comment-14596139 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596147#comment-14596147 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596144#comment-14596144 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13925) Use zookeeper multi to clear znodes in ZKProcedureUtil
[ https://issues.apache.org/jira/browse/HBASE-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596227#comment-14596227 ] Andrew Purtell commented on HBASE-13925: Thanks for the updated patch fixing the watcher issue. The latest precommit failures could be from a noisy build, let me apply and run the tests locally. Back in a bit. Use zookeeper multi to clear znodes in ZKProcedureUtil -- Key: HBASE-13925 URL: https://issues.apache.org/jira/browse/HBASE-13925 Project: HBase Issue Type: Improvement Affects Versions: 0.98.13 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-13925-v1-again.patch, HBASE-13925-v1.patch, HBASE-13925.patch Address the TODO in ZKProcedureUtil clearChildZNodes() and clearZNodes methods -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13147) Load actual META table descriptor, don't use statically defined one.
[ https://issues.apache.org/jira/browse/HBASE-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13147: Fix Version/s: 1.3.0 2.0.0 Load actual META table descriptor, don't use statically defined one. Key: HBASE-13147 URL: https://issues.apache.org/jira/browse/HBASE-13147 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 2.0.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Fix For: 2.0.0, 1.3.0 Attachments: HBASE-13147-branch-1.patch, HBASE-13147-branch-1.v2.patch, HBASE-13147.patch, HBASE-13147.v2.patch, HBASE-13147.v3.patch, HBASE-13147.v4.patch, HBASE-13147.v4.patch, HBASE-13147.v5.patch, HBASE-13147.v6.patch, HBASE-13147.v7.patch In HBASE-13087 stumbled on the fact, that region servers don't see actual meta descriptor, they use their own, statically compiled. Need to fix that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13944) Prefix_Tree seeks to null if the seeked Cell is greater than the last key in the file
[ https://issues.apache.org/jira/browse/HBASE-13944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596229#comment-14596229 ] Anoop Sam John commented on HBASE-13944: The contract in interface says abt the return value for seek(Cell) and the result of call to next(). But not getKeyValue() after the call to seek(). The caller expected to call next() after doing the seek (?) Prefix_Tree seeks to null if the seeked Cell is greater than the last key in the file - Key: HBASE-13944 URL: https://issues.apache.org/jira/browse/HBASE-13944 Project: HBase Issue Type: Bug Components: prefixtree, Scanners Affects Versions: 1.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 2.0.0 When using any DBE other than Prefix_Tree when the internal scanners seek to a key greater than the last key in the file, when we do scanner.getKeyValue() we get the last key in the file, whereas PrefixTree seeks to null. This is a behaviour change. May be in the actual scan case we may not end up in this scenario, need to check with a testcase. But the test in TestSeekTo.testSeekTo() has a clear illustration of this problem. Will take this up after the current activities. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13927) Allow hbase-daemon.sh to conditionally redirect the log or not
[ https://issues.apache.org/jira/browse/HBASE-13927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596248#comment-14596248 ] Andrew Purtell commented on HBASE-13927: +1 Allow hbase-daemon.sh to conditionally redirect the log or not -- Key: HBASE-13927 URL: https://issues.apache.org/jira/browse/HBASE-13927 Project: HBase Issue Type: Bug Components: shell Affects Versions: 2.0.0, 1.2.0 Reporter: Elliott Clark Assignee: Elliott Clark Labels: shell Attachments: HBASE-13927.patch, HBASE-13927.patch Kind of like HBASE_NOEXEC allow hbase-daemon to skip redirecting to a log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12895) Introduce BufferedTable
[
https://issues.apache.org/jira/browse/HBASE-12895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-12895:
Fix Version/s: (was: 1.2.0)
1.3.0
Introduce BufferedTable
---
Key: HBASE-12895
URL: https://issues.apache.org/jira/browse/HBASE-12895
Project: HBase
Issue Type: Improvement
Components: Client, Usability
Reporter: Nick Dimiduk
Fix For: 2.0.0, 1.3.0
Over on HBASE-12728 we extract the feature previously known as disabled
autoFlush into a separate interface called {{BufferedMutator}}. This
interface is like {{Table}} only exposes mutation operations. It would be
useful to provide an adapter that wraps up a {{BufferedMutator}} instance,
providing the rest of the {{Table}} interface API. That way, it's easier for
existing code to drop-in the new API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12890: Fix Version/s: (was: 1.2.0) 1.3.0 bumped to 1.3. Still working on this [~churromorales]? Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12629) Remove hbase.regionsizecalculator.enable from RegionSizeCalculator
[ https://issues.apache.org/jira/browse/HBASE-12629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12629: Fix Version/s: (was: 1.2.0) Status: Open (was: Patch Available) Remove hbase.regionsizecalculator.enable from RegionSizeCalculator -- Key: HBASE-12629 URL: https://issues.apache.org/jira/browse/HBASE-12629 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 2.0.0 Attachments: 0001-HBASE-12629-Remove-hbase.regionsizecalculator.enable.patch The RegionSizeCalculator has a option to disable it. It is on by default and end-to-end use with it disabled is not tested or used anywhere except for a simple unit test. This removes it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596279#comment-14596279
]
Sean Busbey commented on HBASE-13938:
-
(thanks for the heads up, sounds good)
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-10974) Improve DBEs read performance by avoiding byte array deep copies for key[] and value[]
[ https://issues.apache.org/jira/browse/HBASE-10974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-10974: Fix Version/s: (was: 1.2.0) 1.3.0 Status: Open (was: Patch Available) Improve DBEs read performance by avoiding byte array deep copies for key[] and value[] -- Key: HBASE-10974 URL: https://issues.apache.org/jira/browse/HBASE-10974 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.99.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0, 1.3.0 Attachments: HBASE-10974_1.patch As part of HBASE-10801, we tried to reduce the copy of the value [] in forming the KV from the DBEs. The keys required copying and this was restricting us in using Cells and always wanted to copy to be done. The idea here is to replace the key byte[] as ByteBuffer and create a consecutive stream of the keys (currently the same byte[] is used and hence the copy). Use offset and length to track this key bytebuffer. The copy of the encoded format to normal Key format is definitely needed and can't be avoided but we could always avoid the deep copy of the bytes to form a KV and thus use cells effectively. Working on a patch, will post it soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596155#comment-14596155 ] Nick Dimiduk commented on HBASE-13933: -- Thanks Ram. DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13933_0.98.patch, HBASE-13933_2.patch, HBASE-13933_branch-1.1.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13602) Add an option to fail wal recovery when lease recovery fails
[ https://issues.apache.org/jira/browse/HBASE-13602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13602: Fix Version/s: (was: 1.2.0) 1.3.0 Add an option to fail wal recovery when lease recovery fails Key: HBASE-13602 URL: https://issues.apache.org/jira/browse/HBASE-13602 Project: HBase Issue Type: Improvement Components: wal Reporter: Sean Busbey Labels: operability Fix For: 2.0.0, 1.3.0 Currently, if lease recovery doesn't succeed over an extended timeout (default 15 minutes), then we issue a log message about possible data loss and continue with recovering the edits in that file. In some deployments this potential for dataloss might be unacceptable. In those situations it would be good to have a configurable setting that marks the recovery failed instead. Should default to off (at least in branch-1) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13900) duplicate methods between ProtobufMagic and ProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596219#comment-14596219 ] Andrew Purtell edited comment on HBASE-13900 at 6/22/15 4:56 PM: - [~misty] did the necessary revert and recommit already. Thanks [~misty]! was (Author: apurtell): [~misty] did the necessary revert an recommit already. Thanks [~misty]! duplicate methods between ProtobufMagic and ProtobufUtil Key: HBASE-13900 URL: https://issues.apache.org/jira/browse/HBASE-13900 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Assignee: Gabor Liptak Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13900.1.patch Several methods duplicated between: hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufMagic.java and hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java isPBMagicPrefix() isPBMagicPrefix() lengthOfPBMagic() ProtobufUtil partically references ProtobufMagic, but it also has different compare implementation ... Maybe this was based on packaging/signature need? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596142#comment-14596142 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596146#comment-14596146 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596141#comment-14596141 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596138#comment-14596138 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596148#comment-14596148 ] Talat UYARER commented on HBASE-11819: -- Ok [~busbey] ASAP I will create new patch for master and branch-1 Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13017) Backport HBASE-12035 Keep table state in Meta to branch-1
[ https://issues.apache.org/jira/browse/HBASE-13017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-13017: Fix Version/s: (was: 1.2.0) 1.3.0 bumping from 1.2. Still working on this [~octo47]? Backport HBASE-12035 Keep table state in Meta to branch-1 - Key: HBASE-13017 URL: https://issues.apache.org/jira/browse/HBASE-13017 Project: HBase Issue Type: Improvement Components: master Affects Versions: 1.1.0 Reporter: Andrey Stepachev Assignee: Andrey Stepachev Labels: backport Fix For: 1.3.0 Attachments: HBASE-13017-branch-1.patch, HBASE-13017-branch-1.v1.patch, HBASE-13017-branch-1.v1.patch, HBASE-13017-branch-1.v2.patch, HBASE-13017-branch-1.v3.patch, HBASE-13017-branch-1.v4.patch, HBASE-13017-branch-1.v5.patch, HBASE-13017-branch-1.v6.patch Lets backport that feature to branch-1.0 adapting HBASE-12035 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13936) Improve configuration framework
[ https://issues.apache.org/jira/browse/HBASE-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13936: --- Attachment: DynamicConfigs.v01.docx Here's the Google doc downloaded in Word format, for posterity. Improve configuration framework --- Key: HBASE-13936 URL: https://issues.apache.org/jira/browse/HBASE-13936 Project: HBase Issue Type: Umbrella Reporter: Apekshit Sharma Attachments: DynamicConfigs.v01.docx, design.png Here's the design doc: https://docs.google.com/document/d/1WiO2bqguR2DaVT-J2SZTCONbQ3pEhpbOI_bbLMaXRjE/edit# Main changes: get*(foo.bar, default_value) --- get*(HConfig.FOO_BAR) // using enums Robust framework and better documentation for dynamic configurations. Basic overview of new design: !design.png! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12217) System load average based client pushback
[ https://issues.apache.org/jira/browse/HBASE-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-12217: Fix Version/s: (was: 1.2.0) 1.3.0 I think we'd be interested in more than just load average if it was available. Over in HBASE-13103 [~lhofhansl] calls out several other metrics we'd want to know about when auto-tuning number of regions. System load average based client pushback - Key: HBASE-12217 URL: https://issues.apache.org/jira/browse/HBASE-12217 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 2.0.0, 1.3.0 If a RegionServer host is already heavily loaded* then it might not be best to accept more work in the form of coprocessor invocations. This could generalize to all RPC work, perhaps as part of a broader admission control initiative, but I think it makes sense to start small in an obvious place. *: We could use % CPU utilization or the UNIX 1min or 5min load average to determine this, and provide an option for choosing between those alternatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13923) Loaded region coprocessor are not reported in shell status commands
[
https://issues.apache.org/jira/browse/HBASE-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596101#comment-14596101
]
Hadoop QA commented on HBASE-13923:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12741028/HBASE-13923.patch
against master branch at commit f9b17bfd37624374904ec15b9b1300fe4ae0d8c7.
ATTACHMENT ID: 12741028
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.coprocessor.TestClassLoading
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14501//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14501//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14501//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14501//console
This message is automatically generated.
Loaded region coprocessor are not reported in shell status commands
---
Key: HBASE-13923
URL: https://issues.apache.org/jira/browse/HBASE-13923
Project: HBase
Issue Type: Bug
Components: regionserver, shell
Affects Versions: 1.1.0.1
Reporter: Lars George
Assignee: Ashish Singhi
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13923.patch
I added a CP to a table using the shell's alter command. Now I tried to check
if it was loaded (short of resorting to parsing the logs). I recalled the
refguide mentioned the {{status 'detailed'}} command, and tried that to no
avail.
The UI shows the loaded class in the Software Attributes section, so the info
is there. But a shell status command (even after waiting 12+ hours shows
nothing. Here an example of a server that has it loaded according to
{{describe}} and the UI, but the shell lists this:
{noformat}
slave-1.internal.larsgeorge.com:16020 1434486031598
requestsPerSecond=0.0, numberOfOnlineRegions=5, usedHeapMB=278,
maxHeapMB=941, numberOfStores=5, numberOfStorefiles=3,
storefileUncompressedSizeMB=2454, storefileSizeMB=2454,
compressionRatio=1., memstoreSizeMB=0, storefileIndexSizeMB=0,
readRequestsCount=32070, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=2086, totalStaticBloomSizeKB=480,
totalCompactingKVs=0, currentCompactedKVs=0, compactionProgressPct=NaN,
coprocessors=[]
testqauat:usertable,,1433747062257.4db0d7d73cbaac45cb8568d5b185e1f2.
numberOfStores=1, numberOfStorefiles=0,
storefileUncompressedSizeMB=0, lastMajorCompactionTimestamp=0,
storefileSizeMB=0, memstoreSizeMB=0, storefileIndexSizeMB=0,
readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, completeSequenceId=-1,
dataLocality=0.0
testqauat:usertable,user0,1433747062257.f7c7fe3c7d26910010f40101b20f8d06.
numberOfStores=1, numberOfStorefiles=0,
storefileUncompressedSizeMB=0, lastMajorCompactionTimestamp=0,
storefileSizeMB=0, memstoreSizeMB=0,
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596185#comment-14596185
]
Sean Busbey commented on HBASE-13103:
-
I'd like to see this in 1.2, but feature freeze is nigh. I'll leave this
targeting until I actually cut the RC this afternoon/evening. Feel free to bump
out to 1.3 if you don't think things will be ready.
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596251#comment-14596251
]
Nick Dimiduk commented on HBASE-13103:
--
Looks great [~mantonov], +1 ship it. Only thing is you're using java language
{{assert}} in a couple places in test; instead use JUnit's {{assertTrue}}, but
that can be fixed on commit.
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11819: Comment: was deleted (was: Ok [~busbey] ASAP I will create new patch for master and branch-1) Unit test for CoprocessorHConnection - Key: HBASE-11819 URL: https://issues.apache.org/jira/browse/HBASE-11819 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Talat UYARER Priority: Minor Labels: beginner Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1 Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Fix Version/s: 1.3.0
1.2.0
2.0.0
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13945:
---
Status: Patch Available (was: Open)
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.1.0.1, 1.0.1.1, 1.1.0, 1.0.1, 0.98.2, 1.0.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_branch-1.1.patch, HBASE-13945_trunk.patch,
HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13945) Prefix_Tree seekBefore() does not work correctly
[
https://issues.apache.org/jira/browse/HBASE-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13945:
---
Attachment: HBASE-13945_0.98_2.patch
Patch for 0.98
Prefix_Tree seekBefore() does not work correctly
Key: HBASE-13945
URL: https://issues.apache.org/jira/browse/HBASE-13945
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 0.98.2, 1.0.1, 1.1.0, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.1.2, 1.3.0, 1.2.1
Attachments: HBASE-13945_0.98.patch, HBASE-13945_0.98_1.patch,
HBASE-13945_0.98_2.patch, HBASE-13945_branch-1.1.patch,
HBASE-13945_trunk.patch, HBASE-13945_trunk_1.patch
This is related to the TestSeekTo test case where the seekBefore() does not
work with Prefix_Tree because of an issue in getFirstKeyInBlock(). In the
trunk and branch-1 changing the return type of getFirstKeyInBlock() from BB
to Cell resolved the problem, but the same cannot be done in 0.98. Hence we
need a change in the KvUtil.copyToNewBuffer API to handle this. Since the
limit is made as the position - in seekBefore when we do
{code}
byte[] firstKeyInCurrentBlock = Bytes.getBytes(firstKey);
{code}
in HFileReaderV2.seekBefore() we end up in an empty byte array and it would
not be the expected one based on which we try to seek to load a new block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11590) use a specific ThreadPoolExecutor
[ https://issues.apache.org/jira/browse/HBASE-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11590: Fix Version/s: (was: 1.2.0) Status: Open (was: Patch Available) use a specific ThreadPoolExecutor - Key: HBASE-11590 URL: https://issues.apache.org/jira/browse/HBASE-11590 Project: HBase Issue Type: Bug Components: Client, Performance Affects Versions: 1.0.0, 2.0.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 2.0.0 Attachments: tp.patch The JDK TPE creates all the threads in the pool. As a consequence, we create (by default) 256 threads even if we just need a few. The attached TPE create threads only if we have something in the queue. On a PE test with replica on, it improved the 99 latency percentile by 5%. Warning: there are likely some race conditions, but I'm posting it here because there is may be an implementation available somewhere we can use, or a good reason not to do that. So feedback welcome as usual. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10147) Canary additions
[ https://issues.apache.org/jira/browse/HBASE-10147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-10147: Fix Version/s: 2.0.0 Bump. Still working on this? Setting target as 2.0 so it's less likely to get lost. Canary additions Key: HBASE-10147 URL: https://issues.apache.org/jira/browse/HBASE-10147 Project: HBase Issue Type: Improvement Reporter: stack Assignee: Gustavo Anatoly Fix For: 2.0.0 Attachments: HBASE-10147-v2.patch, HBASE-10147-v3.patch, HBASE-10147-v4.patch, HBASE-10147-v5.patch, HBASE-10147.patch, HBASE-10147.patch, HBASE-10147.patch, HBASE-10147.patch I've been using the canary to quickly identify the dodgy machine in my cluster. It is useful for this. What would make it better would be: + Rather than saying how long it took to get a region after you have gotten the region, it'd be sweet to log BEFORE you went to get the region the regionname and the server it is on. I ask for this because as is, I have to wait for the canary to timeout which can be a while. + Second ask is that when I pass the -t, that when it fails, it says what it failed against -- what region and hopefully what server location (might be hard). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10154) Add a unit test for Canary tool
[ https://issues.apache.org/jira/browse/HBASE-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-10154: Fix Version/s: (was: 1.2.0) Status: Open (was: Patch Available) bump. Might be best not to wait for HBASE-10147. Add a unit test for Canary tool --- Key: HBASE-10154 URL: https://issues.apache.org/jira/browse/HBASE-10154 Project: HBase Issue Type: Improvement Components: monitoring, test Reporter: takeshi.miao Assignee: takeshi.miao Priority: Minor Fix For: 2.0.0 Attachments: HBASE-10154-trunk-v01.patch, HBASE-10154-trunk-v02.patch, HBASE-10154-trunk-v03.patch Due to HBASE-10108, I am working to come out a unit test for o.h.hbase.tool.Canary to eliminate this kind of issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-8642) [Snapshot] List and delete snapshot by table
[ https://issues.apache.org/jira/browse/HBASE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8642: --- Fix Version/s: (was: 1.2.0) Status: Open (was: Patch Available) bump. [~julian.zhou] are you still interested in working on this? [Snapshot] List and delete snapshot by table Key: HBASE-8642 URL: https://issues.apache.org/jira/browse/HBASE-8642 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.95.2, 0.95.1, 0.95.0, 0.98.0 Reporter: Julian Zhou Assignee: Julian Zhou Priority: Minor Fix For: 2.0.0 Attachments: 8642-trunk-0.95-v0.patch, 8642-trunk-0.95-v1.patch, 8642-trunk-0.95-v2.patch Support list and delete snapshot by table name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7115) [shell] Provide a way to register custom filters with the Filter Language Parser
[
https://issues.apache.org/jira/browse/HBASE-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-7115:
---
Fix Version/s: (was: 1.2.0)
1.3.0
Status: Open (was: Patch Available)
bumping to 1.3. [~adityakishore] are you still working on this? could you
rebase to current master?
[shell] Provide a way to register custom filters with the Filter Language
Parser
Key: HBASE-7115
URL: https://issues.apache.org/jira/browse/HBASE-7115
Project: HBase
Issue Type: Improvement
Components: Filters, shell
Affects Versions: 0.95.2
Reporter: Aditya Kishore
Assignee: Aditya Kishore
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-7115_trunk.patch, HBASE-7115_trunk.patch,
HBASE-7115_trunk_v2.patch
HBASE-5428 added this capability to thrift interface but the configuration
parameter name is thrift specific.
This patch introduces a more generic parameter hbase.user.filters using
which the user defined custom filters can be specified in the configuration
and loaded in any client that needs to use the filter language parser.
The patch then uses this new parameter to register any user specified filters
while invoking the HBase shell.
Example usage: Let's say I have written a couple of custom filters with class
names *{{org.apache.hadoop.hbase.filter.custom.SuperDuperFilter}}* and
*{{org.apache.hadoop.hbase.filter.custom.SilverBulletFilter}}* and I want to
use them from HBase shell using the filter language.
To do that, I would add the following configuration to {{hbase-site.xml}}
{panel}{{property}}
{{ namehbase.user.filters/name}}
{{
value}}*{{SuperDuperFilter}}*{{:org.apache.hadoop.hbase.filter.custom.SuperDuperFilter,}}*{{SilverBulletFilter}}*{{:org.apache.hadoop.hbase.filter.custom.SilverBulletFilter/value}}
{{/property}}{panel}
Once this is configured, I can launch HBase shell and use these filters in my
{{get}} or {{scan}} just the way I would use a built-in filter.
{code}
hbase(main):001:0 scan 't', {FILTER = SuperDuperFilter(true) AND
SilverBulletFilter(42)}
ROW COLUMN+CELL
status column=cf:a,
timestamp=30438552, value=world_peace
1 row(s) in 0. seconds
{code}
To use this feature in any client, the client needs to make the following
function call as part of its initialization.
{code}
ParseFilter.registerUserFilters(configuration);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Attachment: hbase-13938_v3-branch-1.patch
hbase-13938_v3-branch-1.2.patch
hbase-13938_v3-branch-1.0.patch
hbase-13938_v3-0.98.patch
backport patches. FYI [~enis], [~apurtell], [~busbey]
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
hbase-13938_v3-branch-1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596560#comment-14596560
]
Andrew Purtell commented on HBASE-13938:
+1
I'll defer to the performers of the work on correctness.
Skimmed the patch looking for compat issues. The modified interfaces are marked
Private. In addition I tested Apache Phoenix compilation after these changes
are applied using the head of 4.x-HBase-0.98 branch. Compilation succeeded,
unit tests passed.
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
hbase-13938_v3-branch-1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13843) Fix internal constant text in ReplicationManager.java
[ https://issues.apache.org/jira/browse/HBASE-13843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596034#comment-14596034 ] Hudson commented on HBASE-13843: FAILURE: Integrated in HBase-TRUNK #6590 (See [https://builds.apache.org/job/HBase-TRUNK/6590/]) HBASE-13843 Correct typo in ReplicationAdmin (busbey: rev d51a184051d968dc3bdc00b1c9257c0a9e5ff8a6) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java Fix internal constant text in ReplicationManager.java - Key: HBASE-13843 URL: https://issues.apache.org/jira/browse/HBASE-13843 Project: HBase Issue Type: Bug Components: master Affects Versions: 1.1.0 Reporter: Lars George Assignee: Gabor Liptak Priority: Trivial Labels: beginner Fix For: 2.0.0 Attachments: HBASE-13843.1.patch ReplicationAdmin.java: public static final String CFNAME = columnFamlyName”; (sic) Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-8533) HBaseAdmin does not ride over cluster restart
[ https://issues.apache.org/jira/browse/HBASE-8533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8533: --- Fix Version/s: (was: 1.2.0) Status: Open (was: Patch Available) bump. [~julian.zhou] are you still interested in working on this? HBaseAdmin does not ride over cluster restart - Key: HBASE-8533 URL: https://issues.apache.org/jira/browse/HBASE-8533 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.95.0, 0.98.0 Reporter: Julian Zhou Assignee: Julian Zhou Priority: Minor Fix For: 2.0.0 Attachments: 8533-0.95-v1.patch, 8533-trunk-v1.patch, hbase-8533-trunk-v0.patch For Restful servlet (org.apache.hadoop.hbase.rest.Main (0.94), org.apache.hadoop.hbase.rest.RESTServer (trunk)) on Jetty, we need to first explicitly start the service (% ./bin/hbase-daemon.sh start rest -p 8000 ) for application running. Here is a scenario, sometimes, HBase cluster are stopped/started for maintanence, but rest is a seperated standalone process, which binds the HBaseAdmin at construction method. HBase stop/start cause this binding lost for existing rest servlet. Rest servlet still exist to trying on old bound HBaseAdmin until a long time duration later with an Unavailable caught via an IOException caught in such as RootResource. Could we pairwise the HBase service with HBase rest service with some start/stop options? since seems no reason to still keep the rest servlet process after HBase stopped? When HBase restarts, original rest service could not resume to bind to the new HBase service via its old HBaseAdmin reference? So may we stop the rest when hbase stopped, or even if hbase was killed by acident, restart hbase with rest option could detect the old rest process, kill it and start to bind a new one? From this point of view, application rely on rest api in previous scenario could immediately detect it when setting up http connection session instead of wasting a long time to fail back from IOException with Unavailable from rest servlet. Put current options from the discussion history here from Andrew, Stack and Jean-Daniel, 1) create an HBaseAdmin on demand in rest servlet instead of keeping singleton instance; (another possible enhancement for HBase client: automatic reconnection of an open HBaseAdmin handle after a cluster bounce?) 2) pairwise the rest webapp with hbase webui so the rest is always on with HBase serive; 3) add an option for rest service (such as HBASE_MANAGES_REST) in hbase-env.sh, set HBASE_MANAGES_REST to true, the scripts will start/stop the REST server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7218) Rename Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-7218: --- Fix Version/s: (was: 1.2.0) 1.3.0 Rename Snapshot --- Key: HBASE-7218 URL: https://issues.apache.org/jira/browse/HBASE-7218 Project: HBase Issue Type: New Feature Components: snapshots Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 1.3.0 Attachments: HBASE-7218-v0.patch, HBASE-7218-v1.patch Add the ability to rename a snapshot. HBaseAdmin.renameSnapshot(oldName, newName) shell: snapshot_rename 'oldName', 'newName' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13938:
-
Fix Version/s: (was: 2.0.0)
0.98.14
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
hbase-13938_v3-branch-1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-13906:
Release Note: With this patch NeedUnmanagedConnectionException is
effectively treated as non-retryable exception, but for backwards compatibility
purposes we don't throw it directly, but instead wrap into
DoNotRetryIOException when we return error to the caller.
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0, 1.3.0
Attachments: HBASE-13906-branch-1.v1.patch,
HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-v2.patch v2 patch. fixed some naming issues and merge conflicts. ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596685#comment-14596685
]
Sean Busbey commented on HBASE-13906:
-
looks fine to me. Looks like all IA.Private stuff? If there's some place where
these are private implementations but we're changing behavior a client will see
(e.g. when wrapping NUCE in DNRE) please include a release note.
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0, 1.3.0
Attachments: HBASE-13906-branch-1.v1.patch,
HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596699#comment-14596699
]
Mikhail Antonov commented on HBASE-13906:
-
We never expand 'throws' clause of method signatures here, but there's quite a
few places where we throw 'new DNRE(nukeInstance)'. I'll add a note.
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0, 1.3.0
Attachments: HBASE-13906-branch-1.v1.patch,
HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13947) Use MasterServices instead of Server in AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13947: Attachment: HBASE-13947-v0-branch-1.patch Use MasterServices instead of Server in AssignmentManager - Key: HBASE-13947 URL: https://issues.apache.org/jira/browse/HBASE-13947 Project: HBase Issue Type: Improvement Components: master Affects Versions: 1.2.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 1.2.0 Attachments: HBASE-13947-v0-branch-1.patch Working on a patch for branch-1, the AM is using Server instead of MasterServices and does a cast to MasterServices when needed. We should have MasterServices as arg as we do in master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13925) Use zookeeper multi to clear znodes in ZKProcedureUtil
[ https://issues.apache.org/jira/browse/HBASE-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596658#comment-14596658 ] Andrew Purtell commented on HBASE-13925: All tests pass. Use zookeeper multi to clear znodes in ZKProcedureUtil -- Key: HBASE-13925 URL: https://issues.apache.org/jira/browse/HBASE-13925 Project: HBase Issue Type: Improvement Affects Versions: 0.98.13 Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-13925-v1-again.patch, HBASE-13925-v1.patch, HBASE-13925.patch Address the TODO in ZKProcedureUtil clearChildZNodes() and clearZNodes methods -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13906:
-
Attachment: HBASE-13906-branch-1.v1.patch
Looks good to me, but would be good to get a clean buildbot run; reattaching.
You okay with this [~busbey]?
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0, 1.3.0
Attachments: HBASE-13906-branch-1.v1.patch,
HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13906:
-
Fix Version/s: 1.3.0
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0, 1.3.0
Attachments: HBASE-13906-branch-1.v1.patch,
HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596780#comment-14596780
]
Nick Dimiduk commented on HBASE-13937:
--
Hey [~enis] your master patch really doesn't apply; the code you're aiming to
remove is already not there. I think you should try re-creating it.
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch,
hbase-13937_v3-branch-1.1.patch, hbase-13937_v3.patch, hbase-13937_v3.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13948) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-13948: - Status: Patch Available (was: Open) Expand hadoop2 versions built on the pre-commit --- Key: HBASE-13948 URL: https://issues.apache.org/jira/browse/HBASE-13948 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0 Attachments: 13948.patch For the HBase 1.1 line I've been validating builds against the following hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's do the same in pre-commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-13949) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-13949. -- Resolution: Duplicate Sorry, slow-JIRA double-submit with HBASE-13948. Expand hadoop2 versions built on the pre-commit --- Key: HBASE-13949 URL: https://issues.apache.org/jira/browse/HBASE-13949 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0 For the HBase 1.1 line I've been validating builds against the following hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's do the same in pre-commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13948) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596795#comment-14596795 ] Sean Busbey commented on HBASE-13948: - Do we need to check individual patch releases? Shouldn't either first or latest be sufficient? Expand hadoop2 versions built on the pre-commit --- Key: HBASE-13948 URL: https://issues.apache.org/jira/browse/HBASE-13948 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0 Attachments: 13948.patch For the HBase 1.1 line I've been validating builds against the following hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's do the same in pre-commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13948) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-13948: - Attachment: 13948.patch Expand hadoop2 versions built on the pre-commit --- Key: HBASE-13948 URL: https://issues.apache.org/jira/browse/HBASE-13948 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0 Attachments: 13948.patch For the HBase 1.1 line I've been validating builds against the following hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's do the same in pre-commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-13103:
Status: Patch Available (was: Open)
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch, HBASE-13103-v3.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596852#comment-14596852
]
Enis Soztutar commented on HBASE-13938:
---
Did an interdiff between 1.1 and 1.2/1.0 and 0.98 patches. They look good.
In 1.0 patch this is not needed (but it is ignorable):
+ private Random random = new Random();
Did you re-generate the PB files?
bq. I think this is not needed on master because meta updates are run from
master.
Now thinking about it, I think we should at least make the PB changes in
master, otherwise the wire compat might be broken accidentally. Let me put up a
patch for master.
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
hbase-13938_v3-branch-1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596863#comment-14596863
]
Nick Dimiduk commented on HBASE-13938:
--
{quote}
In 1.0 patch this is not needed (but it is ignorable):
+ private Random random = new Random();
{quote}
Ah thanks. I can remove that on commit.
bq. Did you re-generate the PB files?
I *think* I did this on all branches, but I'll be sure to do so again before
pushing; at worst it'll be a no-op.
bq. I think we should at least make the PB changes in master
Yeah, that's a good point.
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch,
hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch,
hbase-13938_v3-branch-1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596887#comment-14596887
]
Nick Dimiduk commented on HBASE-13937:
--
With patch v3, same loop of {{TestDistributedLogSplitting}} on branch-1.1 is
passing consistently for me; +1 stands.
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch,
hbase-13937_v3-branch-1.1.patch, hbase-13937_v3.patch, hbase-13937_v3.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596889#comment-14596889
]
Mikhail Antonov commented on HBASE-13906:
-
TestRB looks unrelated (this needs to be fixed in separate jira I think, often
fails lately)
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0, 1.3.0
Attachments: HBASE-13906-branch-1.v1.patch,
HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13950) Add a NoopProcedureStore for testing
[ https://issues.apache.org/jira/browse/HBASE-13950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-13950: Attachment: HBASE-13950-v0-branch-1.patch Add a NoopProcedureStore for testing Key: HBASE-13950 URL: https://issues.apache.org/jira/browse/HBASE-13950 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0, 1.2.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13950-v0-branch-1.patch Add a NoopProcedureStore and an helper in ProcedureTestingUtil to submitAndWait() a procedure without having to do anything else. This is useful to avoid extra code like in case of TestAssignmentManager.processServerShutdownHandler() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13950) Add a NoopProcedureStore for testing
Matteo Bertozzi created HBASE-13950: --- Summary: Add a NoopProcedureStore for testing Key: HBASE-13950 URL: https://issues.apache.org/jira/browse/HBASE-13950 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0, 1.2.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13950-v0-branch-1.patch Add a NoopProcedureStore and an helper in ProcedureTestingUtil to submitAndWait() a procedure without having to do anything else. This is useful to avoid extra code like in case of TestAssignmentManager.processServerShutdownHandler() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13948) Expand hadoop2 versions built on the pre-commit
[ https://issues.apache.org/jira/browse/HBASE-13948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596805#comment-14596805 ] Sean Busbey commented on HBASE-13948: - oh right. yeah I should know better about that. :) any idea how long 9x cycles will take? might be time to start looking at parallel execution. Expand hadoop2 versions built on the pre-commit --- Key: HBASE-13948 URL: https://issues.apache.org/jira/browse/HBASE-13948 Project: HBase Issue Type: Task Components: build Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0 Attachments: 13948.patch For the HBase 1.1 line I've been validating builds against the following hadoop versions: 2.2.0 2.3.0 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0. Let's do the same in pre-commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13947) Use MasterServices instead of Server in AssignmentManager
[
https://issues.apache.org/jira/browse/HBASE-13947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596845#comment-14596845
]
Hadoop QA commented on HBASE-13947:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12741128/HBASE-13947-v0-branch-1.patch
against branch-1 branch at commit d51a184051d968dc3bdc00b1c9257c0a9e5ff8a6.
ATTACHMENT ID: 12741128
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3827 checkstyle errors (more than the master's current 3826 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14506//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14506//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14506//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14506//console
This message is automatically generated.
Use MasterServices instead of Server in AssignmentManager
-
Key: HBASE-13947
URL: https://issues.apache.org/jira/browse/HBASE-13947
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 1.2.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 1.2.0
Attachments: HBASE-13947-v0-branch-1.patch
Working on a patch for branch-1, the AM is using Server instead of
MasterServices and does a cast to MasterServices when needed. We should have
MasterServices as arg as we do in master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13702) ImportTsv: Add dry-run functionality and log bad rows
[ https://issues.apache.org/jira/browse/HBASE-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-13702: Attachment: HBASE-13702-v3.patch ImportTsv: Add dry-run functionality and log bad rows - Key: HBASE-13702 URL: https://issues.apache.org/jira/browse/HBASE-13702 Project: HBase Issue Type: New Feature Reporter: Apekshit Sharma Assignee: Apekshit Sharma Attachments: HBASE-13702-v2.patch, HBASE-13702-v3.patch, HBASE-13702.patch ImportTSV job skips bad records by default (keeps a count though). -Dimporttsv.skip.bad.lines=false can be used to fail if a bad row is encountered. To be easily able to determine which rows are corrupted in an input, rather than failing on one row at a time seems like a good feature to have. Moreover, there should be 'dry-run' functionality in such kinds of tools, which can essentially does a quick run of tool without making any changes but reporting any errors/warnings and success/failure. To identify corrupted rows, simply logging them should be enough. In worst case, all rows will be logged and size of logs will be same as input size, which seems fine. However, user might have to do some work figuring out where the logs. Is there some link we can show to the user when the tool starts which can help them with that? For the dry run, we can simply use if-else to skip over writing out KVs, and any other mutations, if present. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
