[jira] [Commented] (HBASE-13917) Remove string comparison to identify request priority
[
https://issues.apache.org/jira/browse/HBASE-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593923#comment-14593923
]
Andrew Purtell commented on HBASE-13917:
bq. let's wait a couple of weeks before backporting it on all the other
branches. I may have other changes around this stuff.
Ok, I opened HBASE-13941 for that
Remove string comparison to identify request priority
-
Key: HBASE-13917
URL: https://issues.apache.org/jira/browse/HBASE-13917
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13917-v0.patch
We have a couple of if (methodName.equalsIgnoreCase(scan)) { ScanRequest
req = (ScanRequest)parm; } we can replace that string comparison with an
instanceof
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13941) Backport HBASE-13917 (Remove string comparison to identify request priority) to release branches
Andrew Purtell created HBASE-13941: -- Summary: Backport HBASE-13917 (Remove string comparison to identify request priority) to release branches Key: HBASE-13941 URL: https://issues.apache.org/jira/browse/HBASE-13941 Project: HBase Issue Type: Task Reporter: Andrew Purtell Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1 Backport HBASE-13917 (Remove string comparison to identify request priority) to release branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593931#comment-14593931
]
Devaraj Das commented on HBASE-13938:
-
+1
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593935#comment-14593935
]
Andrew Purtell commented on HBASE-13937:
bq. We will not treat exceptions coming from server ping differently, but
instead will keep retrying to ping.
Reply
Looking at the V2 patch. So we check once if the server is in the dead list and
then proceed to ping. This patch hoists out this check:
{code}
+ synchronized (this.onlineServers) {
+if (this.deadservers.isDeadServer(server)) {
+ return false;
+}
+ }
{code}
that HBASE-13172 put into the ping loop. We retain this change from HBASE-13172:
{code}
@@ -851,13 +858,21 @@ public class ServerManager {
return info != null info.hasServerName()
server.getStartcode() == info.getServerName().getStartCode();
}
+ } catch (RegionServerStoppedException | ServerNotRunningYetException e) {
+if (LOG.isDebugEnabled()) {
+ LOG.debug(Couldn't reach + server, e);
+}
+break;
} catch (IOException ioe) {
-LOG.debug(Couldn't reach + server + , try= +
retryCounter.getAttemptTimes()
- + of + retryCounter.getMaxAttempts(), ioe);
+if (LOG.isDebugEnabled()) {
+ LOG.debug(Couldn't reach + server + , try= +
retryCounter.getAttemptTimes() + of
+ + retryCounter.getMaxAttempts(), ioe);
+}
try {
retryCounter.sleepUntilNextRetry();
} catch(InterruptedException ie) {
Thread.currentThread().interrupt();
+ break;
}
}
}
{code}
that breaks out of the ping loop if we catch RegionServerStoppedException or
ServerNotRunningYetException.
This lgtm for application to 0.98, modulo the multicatch (Java 7+ only) will
need to be converted to equivalent Java 6 idiom.
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593993#comment-14593993
]
Stephen Yuan Jiang commented on HBASE-13935:
[~jerryhe] This is from system test run. After forcing to create the table,
the create table actually succeeded. And the most important thing is that we
need to start master.
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13832) Procedure V2: master fail to start due to WALProcedureStore sync failures when HDFS data nodes count is low
[
https://issues.apache.org/jira/browse/HBASE-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594160#comment-14594160
]
Enis Soztutar commented on HBASE-13832:
---
Thanks Matteo, I'll take a look.
Procedure V2: master fail to start due to WALProcedureStore sync failures
when HDFS data nodes count is low
---
Key: HBASE-13832
URL: https://issues.apache.org/jira/browse/HBASE-13832
Project: HBase
Issue Type: Sub-task
Components: master, proc-v2
Affects Versions: 2.0.0, 1.1.0, 1.2.0
Reporter: Stephen Yuan Jiang
Assignee: Matteo Bertozzi
Attachments: HBASE-13832-v0.patch, HDFSPipeline.java
when the data node 3, we got failure in WALProcedureStore#syncLoop() during
master start. The failure prevents master to get started.
{noformat}
2015-05-29 13:27:16,625 ERROR [WALProcedureStoreSyncThread]
wal.WALProcedureStore: Sync slot failed, abort.
java.io.IOException: Failed to replace a bad datanode on the existing
pipeline due to no more good datanodes being available to try. (Nodes:
current=[DatanodeInfoWithStorage[10.333.444.555:50010,DS-3ced-93f4-47b6-9c23-1426f7a6acdc,DISK],
DatanodeInfoWithStorage[10.222.666.777:50010,DS-f9c983b4-1f10-4d5e-8983-490ece56c772,DISK]],
original=[DatanodeInfoWithStorage[10.333.444.555:50010,DS-3ced-93f4-47b6-9c23-1426f7a6acdc,DISK],
DatanodeInfoWithStorage[10.222.666.777:50010,DS-f9c983b4-1f10-4d5e-8983-
490ece56c772,DISK]]). The current failed datanode replacement policy is
DEFAULT, and a client may configure this via
'dfs.client.block.write.replace-datanode-on-failure.policy' in its
configuration.
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:951)
{noformat}
One proposal is to implement some similar logic as FSHLog: if IOException is
thrown during syncLoop in WALProcedureStore#start(), instead of immediate
abort, we could try to roll the log and see whether this resolve the issue;
if the new log cannot be created or more exception from rolling the log, we
then abort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13941) Backport HBASE-13917 (Remove string comparison to identify request priority) to release branches
[ https://issues.apache.org/jira/browse/HBASE-13941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13941: --- Fix Version/s: (was: 1.2.0) Backport HBASE-13917 (Remove string comparison to identify request priority) to release branches Key: HBASE-13941 URL: https://issues.apache.org/jira/browse/HBASE-13941 Project: HBase Issue Type: Task Reporter: Andrew Purtell Fix For: 0.98.14, 1.0.2, 1.1.1 Backport HBASE-13917 (Remove string comparison to identify request priority) to release branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593947#comment-14593947
]
Nick Dimiduk commented on HBASE-13937:
--
{{TestDistributedLogSplitting}} is passing consistently on my side as well. +1
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593992#comment-14593992
]
Hadoop QA commented on HBASE-13939:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740628/HBASE-13939_1.patch
against master branch at commit 8ae4b374e3fa4cd54b8b83bc857d41f0678ad9b6.
ATTACHMENT ID: 12740628
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14467//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14467//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14467//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14467//console
This message is automatically generated.
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch, HBASE-13939_1.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13921) move converter functions from ProtobufUtil into a helper class
[ https://issues.apache.org/jira/browse/HBASE-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak updated HBASE-13921: - Attachment: HBASE-13921.2.patch move converter functions from ProtobufUtil into a helper class Key: HBASE-13921 URL: https://issues.apache.org/jira/browse/HBASE-13921 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Priority: Minor Attachments: HBASE-13921.1.patch, HBASE-13921.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13938:
--
Attachment: hbase-13938_v3-branch-1.1.patch
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch,
hbase-13938_v3-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[
https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593994#comment-14593994
]
Hadoop QA commented on HBASE-13933:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740700/HBASE-13933_branch-1.1.patch
against branch-1.1 branch at commit 8ae4b374e3fa4cd54b8b83bc857d41f0678ad9b6.
ATTACHMENT ID: 12740700
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14472//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14472//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14472//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14472//console
This message is automatically generated.
DBE's seekBefore with tags corrupts the tag's offset information thus leading
to incorrect results
--
Key: HBASE-13933
URL: https://issues.apache.org/jira/browse/HBASE-13933
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13933.patch, HBASE-13933_2.patch,
HBASE-13933_branch-1.1.patch, HBASE-13993_1.patch
The problem occurs with moveToPrevious() case and incase of tags we copy the
previous pointer's tag info to the current because already decoded the tags.
Will check once again before I post other details. I have a test case to
reproduce the problem. Found this while working with MultibyteBuffers and
verified if this is present in trunk - it is in all branches where we have
tags compression (I suppose) will verify
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13666) book.pdf is not renamed during site build
[
https://issues.apache.org/jira/browse/HBASE-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-13666:
-
Fix Version/s: 0.98.14
book.pdf is not renamed during site build
-
Key: HBASE-13666
URL: https://issues.apache.org/jira/browse/HBASE-13666
Project: HBase
Issue Type: Task
Components: site
Reporter: Nick Dimiduk
Assignee: Gabor Liptak
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13666.1.patch, HBASE-13666.2.patch,
HBASE-13666.3.patch
Noticed this while testing HBASE-13665. Looks like the post-site hook is
broken or not executed, so the file {{book.pdf}} is not copied over to the
expected name {{apache_hbase_reference_guide.pdf}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13214) Remove deprecated and unused methods from HTable class
[
https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594081#comment-14594081
]
Hadoop QA commented on HBASE-13214:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740587/HBASE-13214-v2-again.patch
against master branch at commit 8ae4b374e3fa4cd54b8b83bc857d41f0678ad9b6.
ATTACHMENT ID: 12740587
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 114
new or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.util.TestProcessBasedCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14468//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14468//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14468//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14468//console
This message is automatically generated.
Remove deprecated and unused methods from HTable class
--
Key: HBASE-13214
URL: https://issues.apache.org/jira/browse/HBASE-13214
Project: HBase
Issue Type: Sub-task
Components: API
Affects Versions: 2.0.0
Reporter: Mikhail Antonov
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again.patch,
HBASE-13214-v2.patch, HBASE-13214.patch
Methods like #getRegionLocation(), #isTableEnabled() etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594118#comment-14594118
]
Andrew Purtell commented on HBASE-13937:
bq. Do you mind re-review according to above?
The check for isDeadServer is moved out of the retry loop:
{code}
@@ -903,13 +901,14 @@ public class ServerManager {
public boolean isServerReachable(ServerName server) {
if (server == null) throw new NullPointerException(Passed server is
null);
+synchronized (this.onlineServers) {
+ if (this.deadservers.isDeadServer(server)) {
+return false;
+ }
+}
+
RetryCounter retryCounter = pingRetryCounterFactory.create();
while (retryCounter.shouldRetry()) {
- synchronized (this.onlineServers) {
-if (this.deadservers.isDeadServer(server)) {
- return false;
-}
- }
try {
AdminService.BlockingInterface admin = getRsAdmin(server);
if (admin != null) {
{code}
Yes, I was wrong about the second part. The patch under review here does:
{code}
@@ -917,11 +916,6 @@ public class ServerManager {
return info != null info.hasServerName()
server.getStartcode() == info.getServerName().getStartCode();
}
- } catch (RegionServerStoppedException | ServerNotRunningYetException e) {
-if (LOG.isDebugEnabled()) {
- LOG.debug(Couldn't reach + server, e);
-}
-break;
{code}
I must have gone back to the wrong tab and looked at the patch on the original
issue.
lgtm, FWIW
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13921) move converter functions from ProtobufUtil into a helper class
[ https://issues.apache.org/jira/browse/HBASE-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594168#comment-14594168 ] Gabor Liptak commented on HBASE-13921: -- [~apurtell] yes, there aren't clear lines on what is a converter function ... I ended up with this line count (of course there are lots of stubs in PotobufUtil): 2026 hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufConverter.java 1799 hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java move converter functions from ProtobufUtil into a helper class Key: HBASE-13921 URL: https://issues.apache.org/jira/browse/HBASE-13921 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Priority: Minor Attachments: HBASE-13921.1.patch, HBASE-13921.2.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13900) duplicate methods between ProtobufMagic and ProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13900: --- Resolution: Fixed Fix Version/s: 2.0.0 Assignee: Gabor Liptak Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) duplicate methods between ProtobufMagic and ProtobufUtil Key: HBASE-13900 URL: https://issues.apache.org/jira/browse/HBASE-13900 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Assignee: Gabor Liptak Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13900.1.patch Several methods duplicated between: hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufMagic.java and hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java isPBMagicPrefix() isPBMagicPrefix() lengthOfPBMagic() ProtobufUtil partically references ProtobufMagic, but it also has different compare implementation ... Maybe this was based on packaging/signature need? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13900) duplicate methods between ProtobufMagic and ProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593985#comment-14593985 ] Andrew Purtell edited comment on HBASE-13900 at 6/19/15 9:26 PM: - Not sure what's up with the javadoc warning either, I don't see it. Committed to master. Thanks for the patch [~gliptak]! was (Author: apurtell): Not sure what's up with the javadoc warning either, I don't see it. Committed to mater. Thanks for the patch [~gliptak]! duplicate methods between ProtobufMagic and ProtobufUtil Key: HBASE-13900 URL: https://issues.apache.org/jira/browse/HBASE-13900 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Assignee: Gabor Liptak Priority: Minor Fix For: 2.0.0 Attachments: HBASE-13900.1.patch Several methods duplicated between: hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufMagic.java and hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java isPBMagicPrefix() isPBMagicPrefix() lengthOfPBMagic() ProtobufUtil partically references ProtobufMagic, but it also has different compare implementation ... Maybe this was based on packaging/signature need? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13937:
--
Attachment: hbase-13937_v3.patch
bq. The check for isDeadServer is moved out of the retry loop:
Nice catch. v3 patch moves it back inside the loop (basically not changing that
part).
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch,
hbase-13937_v3.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594163#comment-14594163
]
Enis Soztutar commented on HBASE-13935:
---
[~ndimiduk], do you want this in 1.1.1?
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13921) move converter functions from ProtobufUtil into a helper class
[ https://issues.apache.org/jira/browse/HBASE-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593941#comment-14593941 ] Andrew Purtell commented on HBASE-13921: What do you envision should remain in ProtobufUtil? move converter functions from ProtobufUtil into a helper class Key: HBASE-13921 URL: https://issues.apache.org/jira/browse/HBASE-13921 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Priority: Minor Attachments: HBASE-13921.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Yuan Jiang updated HBASE-13935:
---
Fix Version/s: 1.1.1
1.2.0
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[
https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594052#comment-14594052
]
Hadoop QA commented on HBASE-13448:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740605/HBASE-13448_V5.patch
against master branch at commit 8ae4b374e3fa4cd54b8b83bc857d41f0678ad9b6.
ATTACHMENT ID: 12740605
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestCheckAndMutate
org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint
org.apache.hadoop.hbase.security.visibility.TestVisibilityWithCheckAuths
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol
org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush
org.apache.hadoop.hbase.TestFullLogReconstruction
org.apache.hadoop.hbase.mapreduce.TestWALRecordReader
org.apache.hadoop.hbase.TestLocalHBaseCluster
org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat
org.apache.hadoop.hbase.client.TestSnapshotFromClientWithRegionReplicas
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.mapreduce.TestWALPlayer
org.apache.hadoop.hbase.coprocessor.TestCoprocessorStop
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
org.apache.hadoop.hbase.regionserver.TestClusterId
org.apache.hadoop.hbase.client.TestClientOperationInterrupt
org.apache.hadoop.hbase.coprocessor.TestOpenTableInCoprocessor
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.regionserver.TestScannerRetriableFailure
org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery
org.apache.hadoop.hbase.mapreduce.TestCellCounter
org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed
org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock
org.apache.hadoop.hbase.security.visibility.TestDefaultScanLabelGeneratorStack
org.apache.hadoop.hbase.regionserver.TestRegionServerHostname
org.apache.hadoop.hbase.regionserver.TestCompaction
org.apache.hadoop.hbase.util.TestMergeTool
org.apache.hadoop.hbase.regionserver.TestScannerWithBulkload
org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
org.apache.hadoop.hbase.regionserver.TestTags
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594050#comment-14594050
]
Mikhail Antonov commented on HBASE-13103:
-
bq. Yeah, there should be some upper bound on the total number of regions,
which I assume would be something like $MAX_REGIONS_PER_SERVER * $NUM_SERVERS,
where max regions per server is configurable.
I thought the limit is the other way around, i.e. the total number of regions
is more or less fixed (as assignment manager won't handle that properly),
hence, increasing number of region servers would inevitably decrease number of
regions per RS - larger and larger regions?
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593935#comment-14593935
]
Andrew Purtell edited comment on HBASE-13937 at 6/19/15 8:43 PM:
-
bq. We will not treat exceptions coming from server ping differently, but
instead will keep retrying to ping.
Looking at the V2 patch. So we check once if the server is in the dead list and
then proceed to ping. This patch hoists out this check:
{code}
+ synchronized (this.onlineServers) {
+if (this.deadservers.isDeadServer(server)) {
+ return false;
+}
+ }
{code}
that HBASE-13172 put into the ping loop. We retain this change from HBASE-13172:
{code}
@@ -851,13 +858,21 @@ public class ServerManager {
return info != null info.hasServerName()
server.getStartcode() == info.getServerName().getStartCode();
}
+ } catch (RegionServerStoppedException | ServerNotRunningYetException e) {
+if (LOG.isDebugEnabled()) {
+ LOG.debug(Couldn't reach + server, e);
+}
+break;
} catch (IOException ioe) {
-LOG.debug(Couldn't reach + server + , try= +
retryCounter.getAttemptTimes()
- + of + retryCounter.getMaxAttempts(), ioe);
+if (LOG.isDebugEnabled()) {
+ LOG.debug(Couldn't reach + server + , try= +
retryCounter.getAttemptTimes() + of
+ + retryCounter.getMaxAttempts(), ioe);
+}
try {
retryCounter.sleepUntilNextRetry();
} catch(InterruptedException ie) {
Thread.currentThread().interrupt();
+ break;
}
}
}
{code}
that breaks out of the ping loop if we catch RegionServerStoppedException or
ServerNotRunningYetException, or are interrupted.
This lgtm for application to 0.98, modulo the multicatch (Java 7+ only) will
need to be converted to equivalent Java 6 idiom.
was (Author: apurtell):
bq. We will not treat exceptions coming from server ping differently, but
instead will keep retrying to ping.
Reply
Looking at the V2 patch. So we check once if the server is in the dead list and
then proceed to ping. This patch hoists out this check:
{code}
+ synchronized (this.onlineServers) {
+if (this.deadservers.isDeadServer(server)) {
+ return false;
+}
+ }
{code}
that HBASE-13172 put into the ping loop. We retain this change from HBASE-13172:
{code}
@@ -851,13 +858,21 @@ public class ServerManager {
return info != null info.hasServerName()
server.getStartcode() == info.getServerName().getStartCode();
}
+ } catch (RegionServerStoppedException | ServerNotRunningYetException e) {
+if (LOG.isDebugEnabled()) {
+ LOG.debug(Couldn't reach + server, e);
+}
+break;
} catch (IOException ioe) {
-LOG.debug(Couldn't reach + server + , try= +
retryCounter.getAttemptTimes()
- + of + retryCounter.getMaxAttempts(), ioe);
+if (LOG.isDebugEnabled()) {
+ LOG.debug(Couldn't reach + server + , try= +
retryCounter.getAttemptTimes() + of
+ + retryCounter.getMaxAttempts(), ioe);
+}
try {
retryCounter.sleepUntilNextRetry();
} catch(InterruptedException ie) {
Thread.currentThread().interrupt();
+ break;
}
}
}
{code}
that breaks out of the ping loop if we catch RegionServerStoppedException or
ServerNotRunningYetException.
This lgtm for application to 0.98, modulo the multicatch (Java 7+ only) will
need to be converted to equivalent Java 6 idiom.
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593936#comment-14593936
]
Devaraj Das commented on HBASE-13937:
-
+1
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13921) move converter functions from ProtobufUtil into a helper class
[ https://issues.apache.org/jira/browse/HBASE-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594019#comment-14594019 ] Gabor Liptak commented on HBASE-13921: -- The other non-convertor helper functions (it seems to be an about even split ...). The comment in ProtobufUtil: @InterfaceAudience.Private // TODO: some clients (Hive, etc) use this class hence the goal of keeping the interface unchanged. move converter functions from ProtobufUtil into a helper class Key: HBASE-13921 URL: https://issues.apache.org/jira/browse/HBASE-13921 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Priority: Minor Attachments: HBASE-13921.1.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13931) Move Unsafe based operations to UnsafeAccess
[
https://issues.apache.org/jira/browse/HBASE-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594100#comment-14594100
]
Hadoop QA commented on HBASE-13931:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740608/HBASE-13931_V2.patch
against master branch at commit 8ae4b374e3fa4cd54b8b83bc857d41f0678ad9b6.
ATTACHMENT ID: 12740608
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14474//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14474//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14474//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14474//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14474//console
This message is automatically generated.
Move Unsafe based operations to UnsafeAccess
Key: HBASE-13931
URL: https://issues.apache.org/jira/browse/HBASE-13931
Project: HBase
Issue Type: Improvement
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 2.0.0
Attachments: HBASE-13931.patch, HBASE-13931_V2.patch
We have this new class added. Bytes.java having unsafe based reads in it. Now
HBASE-13916 will add similar unsafe based reads (on BB) in ByteBufferUtils.
We can move the unsafe based operations to UnsafeAccess and otehr places just
refer to here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13940) IntegrationTestBulkLoad needs option to specify output folders used by test
Enis Soztutar created HBASE-13940:
-
Summary: IntegrationTestBulkLoad needs option to specify output
folders used by test
Key: HBASE-13940
URL: https://issues.apache.org/jira/browse/HBASE-13940
Project: HBase
Issue Type: Test
Reporter: Enis Soztutar
Assignee: Rajeshbabu Chintaguntla
{{IntegrationTestBulkLoad}} should allow output directory to be configurable
rather than hard coded dir under {{/user/user.name}}.
Filing on behalf of [~chrajeshbab...@gmail.com].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13900) duplicate methods between ProtobufMagic and ProtobufUtil
[ https://issues.apache.org/jira/browse/HBASE-13900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593985#comment-14593985 ] Andrew Purtell commented on HBASE-13900: Not sure what's up with the javadoc warning either, I don't see it. Committed to mater. Thanks for the patch [~gliptak]! duplicate methods between ProtobufMagic and ProtobufUtil Key: HBASE-13900 URL: https://issues.apache.org/jira/browse/HBASE-13900 Project: HBase Issue Type: Improvement Reporter: Gabor Liptak Priority: Minor Attachments: HBASE-13900.1.patch Several methods duplicated between: hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufMagic.java and hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java isPBMagicPrefix() isPBMagicPrefix() lengthOfPBMagic() ProtobufUtil partically references ProtobufMagic, but it also has different compare implementation ... Maybe this was based on packaging/signature need? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Yuan Jiang updated HBASE-13935:
---
Attachment: HBASE-13935.v1-branch-1.patch
HBASE-13935.v1-branch-1.patch is for both branch-1 and branch-1.1
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594149#comment-14594149
]
Nick Dimiduk commented on HBASE-13103:
--
Yes there's a global upper bound as well, much more discussion over on
HBASE-11165.
I'm talking about a local upper bound on an individual region server's ability
to maintain active regions online for reads and writes. Usually this is
confined by memory pressure due to active memstores of open regions.
I dunno though. Maybe [~lhofhansl] and [~phobos182] want to meter this upper
bound differently?
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593897#comment-14593897
]
Enis Soztutar commented on HBASE-13938:
---
bq. Don't we need the max computation in the
RegionMergeTransaction.prepareMutationsForMerge as well...
Nice catch. v3 handles that as well.
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Enis Soztutar
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt, hbase-13938_v2-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13897) OOM occurs when Import importing a row that including too much KeyValue
[ https://issues.apache.org/jira/browse/HBASE-13897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13897: --- Fix Version/s: (was: 0.98.13) 0.98.14 OOM occurs when Import importing a row that including too much KeyValue Key: HBASE-13897 URL: https://issues.apache.org/jira/browse/HBASE-13897 Project: HBase Issue Type: Bug Affects Versions: 0.98.13 Reporter: Liu Junhong Assignee: Ted Yu Fix For: 0.98.14 Attachments: HBASE-13897-0.98.patch When importing a row with too many KeyValues (may have too many columns or versions)，KeyValueReducer will incur OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[
https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594010#comment-14594010
]
Nick Dimiduk commented on HBASE-13933:
--
Pushed to master and the branch-1's. {{mvn clean test
-Dtest=org.apache.hadoop.hbase.io.*}} passed everywhere I pushed. Need more
work to backport to 0.98. There's other fields we're not copying from
{{nextState}} so I want to spend some more time with it.
DBE's seekBefore with tags corrupts the tag's offset information thus leading
to incorrect results
--
Key: HBASE-13933
URL: https://issues.apache.org/jira/browse/HBASE-13933
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13933.patch, HBASE-13933_2.patch,
HBASE-13933_branch-1.1.patch, HBASE-13993_1.patch
The problem occurs with moveToPrevious() case and incase of tags we copy the
previous pointer's tag info to the current because already decoded the tags.
Will check once again before I post other details. I have a test case to
reproduce the problem. Found this while working with MultibyteBuffers and
verified if this is present in trunk - it is in all branches where we have
tags compression (I suppose) will verify
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[
https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594113#comment-14594113
]
Hadoop QA commented on HBASE-13614:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740698/HBASE-13614_2.patch
against master branch at commit 8ae4b374e3fa4cd54b8b83bc857d41f0678ad9b6.
ATTACHMENT ID: 12740698
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.1 2.5.2 2.6.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.phoenix.mapreduce.IndexToolIT.testSecondaryIndex(IndexToolIT.java:131)
at
org.apache.phoenix.mapreduce.IndexToolIT.testImmutableGlobalIndex(IndexToolIT.java:74)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14471//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14471//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14471//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14471//console
This message is automatically generated.
Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
--
Key: HBASE-13614
URL: https://issues.apache.org/jira/browse/HBASE-13614
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 2.0.0
Attachments: HBASE-13614.patch, HBASE-13614_1.patch,
HBASE-13614_2.patch
As part of HBASE-10800, move to new CellComparator, we are temp creating a
Cell out of byte[]s so that the Comparator can compare. In read hot path, we
can try minimize the object creations. The parent Jira added some such cases,
which we can solve. This Jira will solve all such cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594090#comment-14594090
]
Enis Soztutar commented on HBASE-13937:
---
bq. Looking at the V2 patch. So we check once if the server is in the dead list
and then proceed to ping. This patch hoists out this check:
I think the patch does the exact opposite. It keeps the {{synchronized
(this.onlineServers) }} part, but removes the
{{catch (RegionServerStoppedException | ServerNotRunningYetException e)}} part.
The intent is to apply v2 directly without reverting the prev patch.
bq. This lgtm for application to 0.98, modulo the multicatch (Java 7+ only)
will need to be converted to equivalent Java 6 idiom.
Do you mind re-review according to above?
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594110#comment-14594110
]
Enis Soztutar commented on HBASE-13935:
---
The patches look good. branch-1 patch is also for master, right? We should
commit to master as well I think.
For 1.0 and before, this can happen if create table handler.prepare created the
znode, but .process() was not able to create the HDFS dir. We see this often,
for example if HDFS perms are broken, and HBase is started. Subsequent starts
will fail because we've already created the znode.
With 1.1+, it should not happen in theory, but did not check whether the
bootstrap code (which creates hbase:namespace) waits for previously executing
procedures to finish. In any case, it is ok to err on the safe side I think.
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594157#comment-14594157
]
Duo Zhang commented on HBASE-13937:
---
{quote}
thus guaranteeing that the region server cannot accept any more writes.
{quote}
But still readable right? Does HBase guarantee this level of consistency?
1. A writes row to HBase.
2 A tells B to read the row.
3. B can read the row from HBase.
If not, then I think remove recoverLease is enough here. Otherwise we still to
make sure that the regionserver can not process any request.
And I think this discussion should be in the parent issue, so +1 on patch v3 :)
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch,
hbase-13937_v3.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594203#comment-14594203
]
Stephen Yuan Jiang commented on HBASE-13935:
[~enis] This JIRA is only for HBASE 1.x and lower releases. We had logic of
first check META and then check znode. The issue happens when table not in
META , but in znode as ENABLED/ENABLING.
This is NOT a issue in master, as we don't use zookeeper to track table state.
The table state is tracked in META table - so we only check META.
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594251#comment-14594251 ] Hudson commented on HBASE-13933: FAILURE: Integrated in HBase-TRUNK #6584 (See [https://builds.apache.org/job/HBase-TRUNK/6584/]) HBASE-13933 DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results (ramkrishna.s.vasudevan) (ndimiduk: rev db08013ebeeaa85802d9795cc72b4c29c5338a47) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13933_2.patch, HBASE-13933_branch-1.1.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594297#comment-14594297 ] Hudson commented on HBASE-13933: FAILURE: Integrated in HBase-1.0 #968 (See [https://builds.apache.org/job/HBase-1.0/968/]) HBASE-13933 DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results (ramkrishna.s.vasudevan) (ndimiduk: rev 5685b6ed49539a463a23f1639448fd2f4e82bd9c) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13933_2.patch, HBASE-13933_branch-1.1.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-13937:
--
Attachment: hbase-13937_v3.patch
reattach.
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch,
hbase-13937_v3.patch, hbase-13937_v3.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13926: --- Attachment: HBASE-13926_V2.patch Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch, HBASE-13926_V2.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13926: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch, HBASE-13926_V2.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13926) Close the scanner only after Call#setResponse
[ https://issues.apache.org/jira/browse/HBASE-13926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13926: --- Status: Open (was: Patch Available) Close the scanner only after Call#setResponse - Key: HBASE-13926 URL: https://issues.apache.org/jira/browse/HBASE-13926 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13926.patch, HBASE-13926_V2.patch This is for HBASE-12295. We will be delivering cells directly from shared cache memory. HBASE-12295 adds a ref count based prevention mechanism to avoid block eviction, when that memory area in use by scanners. We will decrement ref count at scanner close. The memory area will be in use till a cellblock is created or PB scan reply message is built. So we can delay the close of the scanner until the response is set for the scanner Call. This is done via a RpcCallback way. The callback is set on Call at scan time. Once the setResponse is done, the Call will execute the RpcCallback. This jira also adds a facility to do some cleanup/close during the course of scan. Scan from client makes many RPCs fetching N rows each time. Only at the end the scanner close will happen. We will add a new batchClose() facility with which we can do any cleanup after every rpc call is executed and rows fetched for return. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594226#comment-14594226 ] Mikhail Antonov commented on HBASE-13214: - All failures look unrelated to me. Retry again? Once we have good hadoop-qa run, v2 patch looks good to me for master. Pinging [~stack], [~ndimiduk]? Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-13103:
Attachment: HBASE-13103-v2.patch
Here's patch w/ comments from RB addressed, also fixed on-cluster test and
added testcase to check that small but non-adjacent regions aren't attempted to
merge.
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-13103:
Status: Patch Available (was: Open)
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-13103:
Status: Open (was: Patch Available)
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch,
HBASE-13103-v2.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13898) correct additional javadoc failures under java 8
[ https://issues.apache.org/jira/browse/HBASE-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Liptak updated HBASE-13898: - Attachment: HBASE-13898-amendment.1.patch correct additional javadoc failures under java 8 Key: HBASE-13898 URL: https://issues.apache.org/jira/browse/HBASE-13898 Project: HBase Issue Type: Sub-task Components: documentation Affects Versions: 1.2.0 Reporter: Sean Busbey Assignee: Gabor Liptak Priority: Minor Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13898-amendment.1.patch, HBASE-13898.1.patch Java 8's javadoc gives an error where previous javadoc tools gave a more lenient WARN. HBASE-13569 took down the majority of the gaps, but while that issue was in review we gained some more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13898) correct additional javadoc failures under java 8
[ https://issues.apache.org/jira/browse/HBASE-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594293#comment-14594293 ] Gabor Liptak commented on HBASE-13898: -- Uploaded the stragglers ... Please review HBASE-13920 fix for protobuf generated files exclusion. correct additional javadoc failures under java 8 Key: HBASE-13898 URL: https://issues.apache.org/jira/browse/HBASE-13898 Project: HBase Issue Type: Sub-task Components: documentation Affects Versions: 1.2.0 Reporter: Sean Busbey Assignee: Gabor Liptak Priority: Minor Fix For: 2.0.0, 1.2.0 Attachments: HBASE-13898-amendment.1.patch, HBASE-13898.1.patch Java 8's javadoc gives an error where previous javadoc tools gave a more lenient WARN. HBASE-13569 took down the majority of the gaps, but while that issue was in review we gained some more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594302#comment-14594302 ] Hudson commented on HBASE-13933: SUCCESS: Integrated in HBase-1.1 #551 (See [https://builds.apache.org/job/HBase-1.1/551/]) HBASE-13933 DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results (ramkrishna.s.vasudevan) (ndimiduk: rev f023b11d5ccf8b7da5c5285d4b6398130e9a61ed) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13933_2.patch, HBASE-13933_branch-1.1.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13666) book.pdf is not renamed during site build
[
https://issues.apache.org/jira/browse/HBASE-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594388#comment-14594388
]
Hudson commented on HBASE-13666:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #985 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/985/])
HBASE-13666 Correct PDF renaming and bump version of maven-antrun-plugin
(ndimiduk: rev 68bdf873e8cb36d18fad81b4732ed50f144744be)
* dev-support/make_rc.sh
book.pdf is not renamed during site build
-
Key: HBASE-13666
URL: https://issues.apache.org/jira/browse/HBASE-13666
Project: HBase
Issue Type: Task
Components: site
Reporter: Nick Dimiduk
Assignee: Gabor Liptak
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13666.1.patch, HBASE-13666.2.patch,
HBASE-13666.3.patch
Noticed this while testing HBASE-13665. Looks like the post-site hook is
broken or not executed, so the file {{book.pdf}} is not copied over to the
expected name {{apache_hbase_reference_guide.pdf}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13448: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 2.0.0 Attachments: 13448-0.98.txt, HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, HBASE-13448_V4.patch, HBASE-13448_V5.patch, HBASE-13448_V5.patch, HBASE-13448_V5.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13448: --- Attachment: HBASE-13448_V5.patch Attaching same patch for a QA run. Last time all tests failed.. Seems not able to load some HTrace classes! retry.. New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 2.0.0 Attachments: 13448-0.98.txt, HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, HBASE-13448_V4.patch, HBASE-13448_V5.patch, HBASE-13448_V5.patch, HBASE-13448_V5.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13931) Move Unsafe based operations to UnsafeAccess
[ https://issues.apache.org/jira/browse/HBASE-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13931: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) Move Unsafe based operations to UnsafeAccess Key: HBASE-13931 URL: https://issues.apache.org/jira/browse/HBASE-13931 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13931.patch, HBASE-13931_V2.patch, HBASE-13931_V3.patch We have this new class added. Bytes.java having unsafe based reads in it. Now HBASE-13916 will add similar unsafe based reads (on BB) in ByteBufferUtils. We can move the unsafe based operations to UnsafeAccess and otehr places just refer to here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13931) Move Unsafe based operations to UnsafeAccess
[ https://issues.apache.org/jira/browse/HBASE-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13931: --- Attachment: HBASE-13931_V3.patch Move Unsafe based operations to UnsafeAccess Key: HBASE-13931 URL: https://issues.apache.org/jira/browse/HBASE-13931 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13931.patch, HBASE-13931_V2.patch, HBASE-13931_V3.patch We have this new class added. Bytes.java having unsafe based reads in it. Now HBASE-13916 will add similar unsafe based reads (on BB) in ByteBufferUtils. We can move the unsafe based operations to UnsafeAccess and otehr places just refer to here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13931) Move Unsafe based operations to UnsafeAccess
[ https://issues.apache.org/jira/browse/HBASE-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13931: --- Status: Open (was: Patch Available) Move Unsafe based operations to UnsafeAccess Key: HBASE-13931 URL: https://issues.apache.org/jira/browse/HBASE-13931 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13931.patch, HBASE-13931_V2.patch, HBASE-13931_V3.patch We have this new class added. Bytes.java having unsafe based reads in it. Now HBASE-13916 will add similar unsafe based reads (on BB) in ByteBufferUtils. We can move the unsafe based operations to UnsafeAccess and otehr places just refer to here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13935) Orphaned namespace table ZK node should not prevent master to start
[
https://issues.apache.org/jira/browse/HBASE-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594248#comment-14594248
]
Nick Dimiduk commented on HBASE-13935:
--
go ahead.
Orphaned namespace table ZK node should not prevent master to start
---
Key: HBASE-13935
URL: https://issues.apache.org/jira/browse/HBASE-13935
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 1.0.0, 0.98.13
Reporter: Stephen Yuan Jiang
Assignee: Stephen Yuan Jiang
Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1
Attachments: HBASE-13935.v1-0.98.patch,
HBASE-13935.v1-branch-1.0.patch, HBASE-13935.v1-branch-1.patch
Before we have the state-of-art Procedure V2 feature (HBASE 1.0 release or
older), we frequently see the following issue (orphaned ZK node) that prevent
master to start (at least in testing):
{noformat}
2015-06-16 17:54:36,472 FATAL [master:10.0.0.99:6] master.HMaster:
Unhandled exception. Starting shutdown.
org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at
org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:137)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at
org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at
org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1123)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:947)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:618)
at java.lang.Thread.run(Thread.java:745)
2015-06-16 17:54:36,472 INFO [master:10.0.0.99:6] master.HMaster:
Aborting
{noformat}
The above call trace is from a 0.98.x test run. We saw similar issue in
1.0.x run, too.
The proposed fix is to ignore the zk node and force namespace table creation
to be complete so that master can start successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13214: -- Status: Patch Available (was: Open) Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13214: -- Attachment: HBASE-13214-v2-again-v1.patch Retry Hadoop QA again Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13214: -- Status: Open (was: Patch Available) Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13448: --- Status: Open (was: Patch Available) New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 2.0.0 Attachments: 13448-0.98.txt, HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, HBASE-13448_V4.patch, HBASE-13448_V5.patch, HBASE-13448_V5.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13937) Partially revert HBASE-13172
[
https://issues.apache.org/jira/browse/HBASE-13937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594386#comment-14594386
]
Hadoop QA commented on HBASE-13937:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12740771/hbase-13937_v3.patch
against master branch at commit db08013ebeeaa85802d9795cc72b4c29c5338a47.
ATTACHMENT ID: 12740771
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14480//console
This message is automatically generated.
Partially revert HBASE-13172
-
Key: HBASE-13937
URL: https://issues.apache.org/jira/browse/HBASE-13937
Project: HBase
Issue Type: Sub-task
Components: Region Assignment
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.14, 1.2.0, 1.1.1, 1.3.0
Attachments: hbase-13937_v1.patch, hbase-13937_v2.patch,
hbase-13937_v3.patch, hbase-13937_v3.patch
HBASE-13172 is supposed to fix a UT issue, but causes other problems that
parent jira (HBASE-13605) is attempting to fix.
However, HBASE-13605 patch v4 uncovers at least 2 different issues which are,
to put it mildly, major design flaws in AM / RS.
Regardless of 13605, the issue with 13172 is that we catch
{{ServerNotRunningYetException}} from {{isServerReachable()}} and return
false, which then puts the Server to the {{RegionStates.deadServers}} list.
Once it is in that list, we can still assign and unassign regions to the RS
after it has started (because regular assignment does not check whether the
server is in {{RegionStates.deadServers}}. However, after the first assign
and unassign, we cannot assign the region again since then the check for the
lastServer will think that the server is dead.
It turns out that a proper patch for 13605 is very hard without fixing rest
of broken AM assumptions (see HBASE-13605, HBASE-13877 and HBASE-13895 for a
colorful history). For 1.1.1, I think we should just revert parts of
HBASE-13172 for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: hbase-13895_v1-branch-1.1.patch
Quick patch for 1.1
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Fix For: 1.1.1
Attachments: hbase-13895_v1-branch-1.1.patch
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Status: Patch Available (was: Open)
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13214: -- Attachment: HBASE-13214-v2-again.patch Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593111#comment-14593111 ] Ashish Singhi commented on HBASE-13214: --- Attached v2 patch again to see if jenkins gets kicked atleast now. Here is RB link https://reviews.apache.org/r/35642/ Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: 13938-branch-1.1.txt
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: (was: 13938-branch-1.1.txt)
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Fix For: 1.1.1
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13939:
---
Attachment: HBASE-13939.patch
The changes in this patch if committed will remove the change in KeyValueUtil
done in HBASE-13993 because now every where it is cell based.
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13939:
---
Status: Patch Available (was: Open)
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13448) New Cell implementation with cached component offsets/lengths
[ https://issues.apache.org/jira/browse/HBASE-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13448: --- Attachment: HBASE-13448_V5.patch Attaching same patch for a QA run. New Cell implementation with cached component offsets/lengths - Key: HBASE-13448 URL: https://issues.apache.org/jira/browse/HBASE-13448 Project: HBase Issue Type: Sub-task Components: Scanners Reporter: Anoop Sam John Assignee: Anoop Sam John Priority: Blocker Fix For: 2.0.0 Attachments: 13448-0.98.txt, HBASE-13448.patch, HBASE-13448_V2.patch, HBASE-13448_V3.patch, HBASE-13448_V4.patch, HBASE-13448_V5.patch, HBASE-13448_V5.patch, gc.png, hits.png This can be extension to KeyValue and can be instantiated and used in read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593217#comment-14593217 ] ramkrishna.s.vasudevan commented on HBASE-13933: Okie. But for the TestSeekTo testcase to pass with Prefix Tree - we need those changes. Otherwise I have to make the test to work without Preifxtree for now. The optimization I agree we can remove it. DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593250#comment-14593250 ] Anoop Sam John commented on HBASE-13933: +1 for making test to work with all other DBEs. You have HBASE-13939 which would alone solve the issue with prefix tree seekBefore. So let us handle it there. DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13933: --- Status: Open (was: Patch Available) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.1.0.1, 1.0.1.1, 0.98.13, 1.1.0, 1.0.1, 1.0.0, 2.0.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13906) Improve handling of NeedUnmanagedConnectionException
[
https://issues.apache.org/jira/browse/HBASE-13906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593296#comment-14593296
]
Mikhail Antonov commented on HBASE-13906:
-
ASF jenkins is down now, will re-run later, the failure was spurious.
[~ndimiduk] - what do you think about the patch?
Improve handling of NeedUnmanagedConnectionException
Key: HBASE-13906
URL: https://issues.apache.org/jira/browse/HBASE-13906
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 1.1.0
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 1.2.0
Attachments: HBASE-13906-branch-1.v1.patch
During my investigation of HBASE-13833, I made this
[comment|https://issues.apache.org/jira/browse/HBASE-13833?focusedCommentId=14585302page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14585302]
{quote}
One other thing: should we somehow handle NeedUnmanagedConnectionException
similarly to DoNotRetryIOException? It's too late to wire them up thusly, but
for the case of bulk load when the bug is expressed, we go through the full
35x retry loop before eventually failing the RPC. This would be applicable to
branch-1.
{quote}
This would apply only to branch-1 as master no longer has unmanaged
connections. Probably we cannot change the inheritance hierarchy due to
compatibility guarantees, but maybe we can do something like everywhere we
look for DNRIOE, we do the same for NUCE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[ https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593364#comment-14593364 ] ramkrishna.s.vasudevan commented on HBASE-13614: There is one more in the blockIndex area. Will check that also. Will upload another patch if that change is needed. Avoid temp KeyOnlyKeyValue temp objects creations in read hot path -- Key: HBASE-13614 URL: https://issues.apache.org/jira/browse/HBASE-13614 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0 Attachments: HBASE-13614.patch As part of HBASE-10800, move to new CellComparator, we are temp creating a Cell out of byte[]s so that the Comparator can compare. In read hot path, we can try minimize the object creations. The parent Jira added some such cases, which we can solve. This Jira will solve all such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[ https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13614: --- Status: Open (was: Patch Available) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path -- Key: HBASE-13614 URL: https://issues.apache.org/jira/browse/HBASE-13614 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0 Attachments: HBASE-13614.patch As part of HBASE-10800, move to new CellComparator, we are temp creating a Cell out of byte[]s so that the Comparator can compare. In read hot path, we can try minimize the object creations. The parent Jira added some such cases, which we can solve. This Jira will solve all such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[ https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593314#comment-14593314 ] ramkrishna.s.vasudevan commented on HBASE-13933: bq.The other difference between all DBEs and Prefix Tree is that when we seek to a key after the last key all the DBEs will point to the last key when we do scanner.getKeyValue. But Prefix_Tree is giving a null. Is it fine to have this behaviourial change? May be in normal scan this will not happen? This can be done in a seperate JIRA. The if condition in the test case is for the above condition. Will raise a seperate JIRA for that. Let it be there. Once we do HBASE-13933 that will be needed. DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results -- Key: HBASE-13933 URL: https://issues.apache.org/jira/browse/HBASE-13933 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 Attachments: HBASE-13933.patch, HBASE-13933_2.patch, HBASE-13993_1.patch The problem occurs with moveToPrevious() case and incase of tags we copy the previous pointer's tag info to the current because already decoded the tags. Will check once again before I post other details. I have a test case to reproduce the problem. Found this while working with MultibyteBuffers and verified if this is present in trunk - it is in all branches where we have tags compression (I suppose) will verify -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593350#comment-14593350
]
Anoop Sam John commented on HBASE-13939:
getFirstKeyInBlock()
We need rename this then? When it says key returning BB or byte[] makes more
sense?
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch, HBASE-13939_1.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[ https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13614: --- Status: Patch Available (was: Open) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path -- Key: HBASE-13614 URL: https://issues.apache.org/jira/browse/HBASE-13614 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0 Attachments: HBASE-13614.patch, HBASE-13614_1.patch As part of HBASE-10800, move to new CellComparator, we are temp creating a Cell out of byte[]s so that the Comparator can compare. In read hot path, we can try minimize the object creations. The parent Jira added some such cases, which we can solve. This Jira will solve all such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13939:
---
Status: Open (was: Patch Available)
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13939:
---
Attachment: HBASE-13939_1.patch
Updated patch. There was one KeyOnlyKV getting created in the binarySearch
using RAW_BYTES_COMPARATOR.
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch, HBASE-13939_1.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593220#comment-14593220
]
Mikhail Antonov commented on HBASE-13103:
-
Yeah. We used to mention here that region has some ideal size and we should
try to get each region to this size, and I think we mentioned that ideal size
might be a fixed fraction of max size or something like that. May'be needs to
be more configurable.
I guess you assume here that every large table is supposed to be spread across
all RSs, and not just some subset (group?) of them? Also, to make sure I
understand right, when you say 250 regions per RS, you mean 250regions of
each table, or across all tables? Also this number of regions per RS.. I
suppose we can derive it dynamically like (max number of regions total in
cluster, as limited by AM performance, see issue about scaling to 1M regions) /
# of RS? Total max number of regions could be set in config,like 100k or 300k?
I'm thinking about roughly same logic for lower and upper ends (for lower end
another implicit threshold would be max size of each region, and for upper
limit I think there should be 2 more guards - 1) should check that total number
of regions doesn't approach the limits of AM and 2) we don't break table into
ridiculously small regions (less than N hdfs blocks?).
[ergonomics] add region size balancing as a feature of master
-
Key: HBASE-13103
URL: https://issues.apache.org/jira/browse/HBASE-13103
Project: HBase
Issue Type: Improvement
Components: Balancer, Usability
Reporter: Nick Dimiduk
Assignee: Mikhail Antonov
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-13103-v0.patch, HBASE-13103-v1.patch
Often enough, folks miss-judge split points or otherwise end up with a
suboptimal number of regions. We should have an automated, reliable way to
reshape or balance a table's region boundaries. This would be for tables
that contain existing data. This might look like:
{noformat}
Admin#reshapeTable(TableName, int numSplits);
{noformat}
or from the shell:
{noformat}
reshape TABLE, numSplits
{noformat}
Better still would be to have a maintenance process, similar to the existing
Balancer that runs AssignmentManager on an interval, to run the above
reshape operation on an interval. That way, the cluster will automatically
self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13933) DBE's seekBefore with tags corrupts the tag's offset information thus leading to incorrect results
[
https://issues.apache.org/jira/browse/HBASE-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593282#comment-14593282
]
Anoop Sam John commented on HBASE-13933:
+1. Good find.
{code}
if (encoding == DataBlockEncoding.PREFIX_TREE) {
321 // TODO : Fix this
322 assertEquals(null, scanner.getKeyValue());
323 } else {
324 assertEquals(k, toRowStr(scanner.getKeyValue()));
325 }
{code}
Now you don't need this in the test. Is this another bug with PREFIX_TREE?
Have any Jira?
DBE's seekBefore with tags corrupts the tag's offset information thus leading
to incorrect results
--
Key: HBASE-13933
URL: https://issues.apache.org/jira/browse/HBASE-13933
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0, 1.0.1, 1.1.0, 0.98.13, 1.0.1.1, 1.1.0.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0
Attachments: HBASE-13933.patch, HBASE-13933_2.patch,
HBASE-13993_1.patch
The problem occurs with moveToPrevious() case and incase of tags we copy the
previous pointer's tag info to the current because already decoded the tags.
Will check once again before I post other details. I have a test case to
reproduce the problem. Found this while working with MultibyteBuffers and
verified if this is present in trunk - it is in all branches where we have
tags compression (I suppose) will verify
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[ https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13614: --- Attachment: HBASE-13614_1.patch Just a minor change in HFileBlockIndex while loadDataBlockWithScanInfo(). We cannot have the nextIndexedKey as instance level because the index reader is once per reader and not once per scanner. Also the nextIndexKey also acts as an reference to the HConstants.NO_INDEXED_KEY. So we have more or less covered all the cases and should be good. Avoid temp KeyOnlyKeyValue temp objects creations in read hot path -- Key: HBASE-13614 URL: https://issues.apache.org/jira/browse/HBASE-13614 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0 Attachments: HBASE-13614.patch, HBASE-13614_1.patch As part of HBASE-10800, move to new CellComparator, we are temp creating a Cell out of byte[]s so that the Comparator can compare. In read hot path, we can try minimize the object creations. The parent Jira added some such cases, which we can solve. This Jira will solve all such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13932) Add mob integrity check in HFilePrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593317#comment-14593317 ] Bhupendra Kumar Jain commented on HBASE-13932: -- Thanks for addressing comments. New patch lgtm. Just one more change , can be done before commit. 1) evictMobFilesIfNecessary(foundMobFiles, 50); foundMobFiles are initalized with 100. So limit should be 100 or initialize with 50 2) Query : As per current patch, cache scope is only per HFile. Can it be across HFile ( Global scope ) ? Add mob integrity check in HFilePrettyPrinter - Key: HBASE-13932 URL: https://issues.apache.org/jira/browse/HBASE-13932 Project: HBase Issue Type: Sub-task Components: mob Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-13932-V2.patch, HBASE-13932.patch We need to know whether a reference cell is dangling in mob. We can add this to HFilePrettyPrinter. We can add a new option to the command, to check the integrity of mob cells either per region or per file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13939:
---
Status: Patch Available (was: Open)
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch, HBASE-13939_1.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593352#comment-14593352
]
ramkrishna.s.vasudevan commented on HBASE-13939:
bq.getFirstKeyInBlock()
firstKVInBlock? But this is only the Key type of a KeyValue right?
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-13939.patch, HBASE-13939_1.patch
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[ https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13614: --- Status: Patch Available (was: Open) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path -- Key: HBASE-13614 URL: https://issues.apache.org/jira/browse/HBASE-13614 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0 Attachments: HBASE-13614.patch As part of HBASE-10800, move to new CellComparator, we are temp creating a Cell out of byte[]s so that the Comparator can compare. In read hot path, we can try minimize the object creations. The parent Jira added some such cases, which we can solve. This Jira will solve all such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13614) Avoid temp KeyOnlyKeyValue temp objects creations in read hot path
[ https://issues.apache.org/jira/browse/HBASE-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-13614: --- Attachment: HBASE-13614.patch Removes the local KVOnlyKV in the HFileReaderImpl and EncodedSeeker. In all these cases the scanner instances are per get or scan. So we could use the same instance KeyOnlyKV to set the key part for comparisons in blockSeek or seekToblock cases. Also removes unused code in Prefixtree. With this and HBASE-13939 we avoid cases where we create lot of KeyOnlyKeyValue in hot read path. Avoid temp KeyOnlyKeyValue temp objects creations in read hot path -- Key: HBASE-13614 URL: https://issues.apache.org/jira/browse/HBASE-13614 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 2.0.0 Attachments: HBASE-13614.patch As part of HBASE-10800, move to new CellComparator, we are temp creating a Cell out of byte[]s so that the Comparator can compare. In read hot path, we can try minimize the object creations. The parent Jira added some such cases, which we can solve. This Jira will solve all such cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13931) Move Unsafe based operations to UnsafeAccess
[ https://issues.apache.org/jira/browse/HBASE-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593377#comment-14593377 ] ramkrishna.s.vasudevan commented on HBASE-13931: +1 Move Unsafe based operations to UnsafeAccess Key: HBASE-13931 URL: https://issues.apache.org/jira/browse/HBASE-13931 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13931.patch, HBASE-13931_V2.patch We have this new class added. Bytes.java having unsafe based reads in it. Now HBASE-13916 will add similar unsafe based reads (on BB) in ByteBufferUtils. We can move the unsafe based operations to UnsafeAccess and otehr places just refer to here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: (was: hbase-13895_v1-branch-1.1.patch)
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed
[
https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj Das updated HBASE-13938:
Attachment: 13938-branch-1.1.txt
Deletes done during the region merge transaction may get eclipsed
-
Key: HBASE-13938
URL: https://issues.apache.org/jira/browse/HBASE-13938
Project: HBase
Issue Type: Bug
Components: master, regionserver
Reporter: Devaraj Das
Fix For: 1.1.1
Attachments: 13938-branch-1.1.txt
Was looking at an issue from our internal testing. It seems the Deletes of
the region rows from the meta done during the merge transaction could be
eclipsed by the Put of a region row that might have happened moments before.
The master logs this for the merge:
{noformat}
2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12]
master.AssignmentManager: Handled MERGED event;
merged=IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.,
region_a=IntegrationTestIngest,a65c,1434631353820.8b911862d7705ac808b8d132d0154c16.,
region_b=IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.,
on ddas-2-5.openstacklocal,16020,1434632778438
{noformat}
One of the regions that got merged got Opened a few seconds back:
{noformat}
2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1]
regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next
sequenceid=182988
{noformat}
The above would have done a Put in the meta.
Looking at the raw scan of the meta, for the new merged region, the creation
timestamp is 1434633226101:
{noformat}
IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.
column=info:regioninfo, timestamp=1434633226101, value={ENCODED =
0927319db6bf5e128e3bec2a420819aa, NAME =
'IntegrationTestIngest,a65c,1434633226681.0927319db6bf5e128e3bec2a420819aa.',
STARTKEY = 'a65c', ENDKEY = 'b328'}
{noformat}
Looking at the raw scan of the meta, the timestamp for the region open of the
already merged region is 1434633226600. This is a little after the merge
transaction's timestamp.
{noformat}
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:seqnumDuringOpen, timestamp=1434633226600,
value=\x00\x00\x00\x00\x00\x02\xCA\xCC
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:server, timestamp=1434633226600,
value=ddas-2-5.openstacklocal:16020
IntegrationTestIngest,acc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec.
column=info:serverstartcode, timestamp=1434633226600, value=1434632778438
{noformat}
We need to fix it so that the merge region transaction also takes the
master's timestamp. Similar to HBASE-13875.
When this happens, clients start to see a row in the meta with an empty
HRegionInfo (this is because the Put done during the region open only updates
the location information but not the HRI, and the HRI deleted during the
merge transaction remains deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13939) Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
[
https://issues.apache.org/jira/browse/HBASE-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-13939:
---
Issue Type: Sub-task (was: Improvement)
Parent: HBASE-13500
Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell
--
Key: HBASE-13939
URL: https://issues.apache.org/jira/browse/HBASE-13939
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
The getFirstKeyInBlock() in HFileReaderImpl is returning a BB. It is getting
used in seekBefore cases. Because we return a BB we create a KeyOnlyKV once
for comparison
{code}
if (reader.getComparator()
.compareKeyIgnoresMvcc(
new KeyValue.KeyOnlyKeyValue(firstKey.array(),
firstKey.arrayOffset(),
firstKey.limit()), key) = 0) {
long previousBlockOffset = seekToBlock.getPrevBlockOffset();
// The key we are interested in
if (previousBlockOffset == -1) {
// we have a 'problem', the key we want is the first of the file.
return false;
}
{code}
And if the compare fails we again create another KeyOnlyKv
{code}
Cell firstKeyInCurrentBlock = new
KeyValue.KeyOnlyKeyValue(Bytes.getBytes(firstKey));
loadBlockAndSeekToKey(seekToBlock, firstKeyInCurrentBlock, true, key,
true);
{code}
So one object will be enough and that can be returned by getFirstKeyInBlock.
Also will be useful when we go with Buffered backed server cell to change in
one place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13214: -- Status: Patch Available (was: Open) Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2-again.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13214) Remove deprecated and unused methods from HTable class
[ https://issues.apache.org/jira/browse/HBASE-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13214: -- Status: Open (was: Patch Available) Remove deprecated and unused methods from HTable class -- Key: HBASE-13214 URL: https://issues.apache.org/jira/browse/HBASE-13214 Project: HBase Issue Type: Sub-task Components: API Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13214-v1.patch, HBASE-13214-v2.patch, HBASE-13214.patch Methods like #getRegionLocation(), #isTableEnabled() etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13931) Move Unsafe based operations to UnsafeAccess
[ https://issues.apache.org/jira/browse/HBASE-13931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13931: --- Attachment: HBASE-13931_V2.patch Attaching patch with comment fix for a QA run. There seems some issues with QA. Move Unsafe based operations to UnsafeAccess Key: HBASE-13931 URL: https://issues.apache.org/jira/browse/HBASE-13931 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 Attachments: HBASE-13931.patch, HBASE-13931_V2.patch We have this new class added. Bytes.java having unsafe based reads in it. Now HBASE-13916 will add similar unsafe based reads (on BB) in ByteBufferUtils. We can move the unsafe based operations to UnsafeAccess and otehr places just refer to here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
