[jira] [Commented] (HBASE-9135) Upgrade hadoop 1 version to 1.2.1 which is stable
[ https://issues.apache.org/jira/browse/HBASE-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730424#comment-13730424 ] ramkrishna.s.vasudevan commented on HBASE-9135: --- +1 on patch. Any more oonsensus required here? Pls feel free to pitch in. Upgrade hadoop 1 version to 1.2.1 which is stable - Key: HBASE-9135 URL: https://issues.apache.org/jira/browse/HBASE-9135 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Richard Ding Attachments: 9135-v1.txt Here is related discussion: http://search-hadoop.com/m/nA71y1kKHDm1/Hadoop+version+1.2.1+%2528stable%2529+releasedsubj=Re+ANNOUNCE+Hadoop+version+1+2+1+stable+released Older hadoop 1 artifacts would be phased out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730426#comment-13730426 ] Himanshu Vashishtha commented on HBASE-8741: Thanks for the review Stack. bq. What else has changed since previous versions? a) The addition of HLogService interface which provides regions sequenceIDs to HLog at time of rolling. b) Using mock HRS in test methods where we want to control log archiving. bq. Is HLogServices the right name for the Interface that has obtainRegionsSequenceId? Should it be SequenceIdsService? Or just SequenceIds? It is HLog that wants the Service, right? It is not a a Service that HLog is providing? I named it a general Service so as in case if we had to add more service methods to it. But yes, it is service that HLog wants. Looking at other places such as RegionServerServices, I think it could be called RegionSequenceIds. I will do the required refactoring. Also, I ran IT (IntegrationTestBigLinkedList, IntegrationTestBigLinkedListWithChaosMonkey) tests on a patched cluster (BigLinkedListTests) last weekend. Mutations on Regions in recovery mode might have same sequenceIDs - Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.2 Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9136) RPC side changes to have a different codec for server to client communication
ramkrishna.s.vasudevan created HBASE-9136: - Summary: RPC side changes to have a different codec for server to client communication Key: HBASE-9136 URL: https://issues.apache.org/jira/browse/HBASE-9136 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 With reference to the mail sent in the dev list, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 We should have a provision such that the codec on the server side could be different from the one on the client side. This would help to remove the tags for security usecases. This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8496) Implement tags and the internals of how a tag should look like
[ https://issues.apache.org/jira/browse/HBASE-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730470#comment-13730470 ] ramkrishna.s.vasudevan commented on HBASE-8496: --- Posted a new RB https://reviews.apache.org/r/13311/. Implement tags and the internals of how a tag should look like -- Key: HBASE-8496 URL: https://issues.apache.org/jira/browse/HBASE-8496 Project: HBase Issue Type: New Feature Affects Versions: 0.98.0, 0.95.2 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Attachments: Comparison.pdf, HBASE-8496_2.patch, HBASE-8496.patch, Tag design.pdf, Tag design_updated.pdf, Tag_In_KV_Buffer_For_reference.patch The intent of this JIRA comes from HBASE-7897. This would help us to decide on the structure and format of how the tags should look like. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9056) Add client side support for Tags
[ https://issues.apache.org/jira/browse/HBASE-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730471#comment-13730471 ] ramkrishna.s.vasudevan commented on HBASE-9056: --- The current patch in the RB https://reviews.apache.org/r/13311/ has this already. I can seperate it out for this JIRA. Add client side support for Tags Key: HBASE-9056 URL: https://issues.apache.org/jira/browse/HBASE-9056 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Clients should now be able to support tags. Also the LoadTestTool and Performance Evaluation tool should work with Tags. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9136) RPC side changes to have a different codec for server to client communication
[ https://issues.apache.org/jira/browse/HBASE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730472#comment-13730472 ] Anoop Sam John commented on HBASE-9136: --- Ram As Stack said in the dev mailing list (Could the codec itself figure its context and act appropriately) that would be better?? Is ur idea that Encoder in the codec at RS does not write tags? Then Replication will be an issue. During Replication teh encoder should write the Tags to the peer cluster RPC side changes to have a different codec for server to client communication - Key: HBASE-9136 URL: https://issues.apache.org/jira/browse/HBASE-9136 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 With reference to the mail sent in the dev list, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 We should have a provision such that the codec on the server side could be different from the one on the client side. This would help to remove the tags for security usecases. This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9136) RPC side changes to have a different codec for server to client communication
[ https://issues.apache.org/jira/browse/HBASE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730483#comment-13730483 ] ramkrishna.s.vasudevan commented on HBASE-9136: --- Yes that would be the idea here. Changing the codec to figure it is what am also thinking. May be the title says different. This JIRA is aimed to provide that capability in the codec itself. I added this line in the description. RPC side changes to have a different codec for server to client communication - Key: HBASE-9136 URL: https://issues.apache.org/jira/browse/HBASE-9136 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 With reference to the mail sent in the dev list, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 We should have a provision such that the codec on the server side could be different from the one on the client side. This would help to remove the tags for security usecases. This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730496#comment-13730496
]
Hadoop QA commented on HBASE-8741:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596281/HBASE-8741-v4-again.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6616//console
This message is automatically generated.
Mutations on Regions in recovery mode might have same sequenceIDs
-
Key: HBASE-8741
URL: https://issues.apache.org/jira/browse/HBASE-8741
Project: HBase
Issue Type: Bug
Components: MTTR
Affects Versions: 0.95.1
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Fix For: 0.95.2
Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch,
HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch,
HBASE-8741-v4.patch
Currently, when opening a region, we find the maximum sequence ID from all
its HFiles and then set the LogSequenceId of the log (in case the later is at
a small value). This works good in recovered.edits case as we are not writing
to the region until we have replayed all of its previous edits.
With distributed log replay, if we want to enable writes while a region is
under recovery, we need to make sure that the logSequenceId maximum
logSequenceId of the old regionserver. Otherwise, we might have a situation
where new edits have same (or smaller) sequenceIds.
We can store region level information in the WALTrailer, than this scenario
could be avoided by:
a) reading the trailer of the last completed file, i.e., last wal file
which has a trailer and,
b) completely reading the last wal file (this file would not have the
trailer, so it needs to be read completely).
In future, if we switch to multi wal file, we could read the trailer for all
completed WAL files, and reading the remaining incomplete files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9132) Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94
[ https://issues.apache.org/jira/browse/HBASE-9132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730515#comment-13730515 ] Matteo Bertozzi commented on HBASE-9132: +1, patch looks good to me Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94 -- Key: HBASE-9132 URL: https://issues.apache.org/jira/browse/HBASE-9132 Project: HBase Issue Type: Improvement Reporter: Dave Latham Assignee: Dave Latham Fix For: 0.94.11 Attachments: HBASE-9132-0.94.patch As described in HBASE-8778, anything that needs to read table descriptors (such as region assignments) currently results in a table directory scan which is slow on tables with many regions. HBASE-8778 moved the table descriptors to a subdir for 0.96+. For 0.94 a smaller change that is safe with rolling restarts is to cache the directory modtime and use the cached table descriptor if the directory modtime has not changed since a table descriptor change always changes the directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9137) Add Tag dictionary in WAL compression
ramkrishna.s.vasudevan created HBASE-9137: - Summary: Add Tag dictionary in WAL compression Key: HBASE-9137 URL: https://issues.apache.org/jira/browse/HBASE-9137 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan We can add tag dictionary like we have one for rowdictionary, familydictionary. But this has to be done after stabilizing HBASE-7391. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730596#comment-13730596
]
ramkrishna.s.vasudevan commented on HBASE-7391:
---
Going thro this again for having WAL compression for tags.
The above said OOME is more likely to happen when we are creating
RecoveredEdits and when we are reading recovered edits.
In case of creating recovered.edits HLog files we would be instantiating
Writers per region and each Writer would create a compression context and hence
the 5 type of dictionaries would be created. In the normal case this could be
ok (where the HLog is instantiated for the RS).
So incase of recovered edits we can easily save on the initalization part. We
know that for the recovered edits the Regionname, tablename are always constant
because the recovered.edits knows for which region and which table it is.
Similarly the family can be moved out of this array which is 1024 in size by
default. Assuming the number of families are not huge generally.
Still need to think on the qualifier for wider tables.
The row dictionary can be as how it is implemented now. Tags also could go
with the same way like rows. For the row and tag part we could identify a
better way to optimize the memory usage.
So having a different implementation of the dictionary as said above for the
recovered.edits creation would directly reduce the memory consumption.
Review/improve HLog compression's memory consumption
Key: HBASE-7391
URL: https://issues.apache.org/jira/browse/HBASE-7391
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Fix For: 0.95.2
From Ram in
http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
{quote}
One small observation after giving +1 on the RC.
The WAL compression feature causes OOME and causes Full GC.
The problem is, if we have 1500 regions and I need to create recovered.edits
for each of the region (I don’t have much data in the regions (~300MB)).
Now when I try to build the dictionary there is a Node object getting
created.
Each node object occupies 32 bytes.
We have 5 such dictionaries.
Initially we create indexToNodes array and its size is 32767.
So now we have 32*5*32767 = ~5MB.
Now I have 1500 regions.
So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
initial memory foot print and this never allows me to split the logs and I
am not able to make the cluster up at all.
Our configured heap size was 8GB, tested in 3 node cluster with 5000
regions, very less data( 1GB in hdfs cluster including replication), some
small data is spread evenly across all regions.
The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
objects)*noofregions.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9138) the function getHaseIntegrationTestingUtility() is a misspelling
chendihao created HBASE-9138: Summary: the function getHaseIntegrationTestingUtility() is a misspelling Key: HBASE-9138 URL: https://issues.apache.org/jira/browse/HBASE-9138 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.8 Reporter: chendihao Priority: Trivial the function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be getHBaseIntegrationTestingUtility() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9138) the function getHaseIntegrationTestingUtility() is a misspelling
[ https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chendihao updated HBASE-9138: - Attachment: ChaosMonkey.java.patch Simply correct the function name the function getHaseIntegrationTestingUtility() is a misspelling Key: HBASE-9138 URL: https://issues.apache.org/jira/browse/HBASE-9138 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.8 Reporter: chendihao Priority: Trivial Attachments: ChaosMonkey.java.patch the function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be getHBaseIntegrationTestingUtility() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9138) the function getHaseIntegrationTestingUtility() is a misspelling
[ https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730646#comment-13730646 ] chendihao commented on HBASE-9138: -- patch for 0.94.x the function getHaseIntegrationTestingUtility() is a misspelling Key: HBASE-9138 URL: https://issues.apache.org/jira/browse/HBASE-9138 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.8 Reporter: chendihao Priority: Trivial Attachments: ChaosMonkey.java.patch the function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be getHBaseIntegrationTestingUtility() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9138) the name of function getHaseIntegrationTestingUtility() is a misspelling
[ https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chendihao updated HBASE-9138: - Summary: the name of function getHaseIntegrationTestingUtility() is a misspelling (was: the function getHaseIntegrationTestingUtility() is a misspelling) the name of function getHaseIntegrationTestingUtility() is a misspelling Key: HBASE-9138 URL: https://issues.apache.org/jira/browse/HBASE-9138 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.8 Reporter: chendihao Priority: Trivial Attachments: ChaosMonkey.java.patch the function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be getHBaseIntegrationTestingUtility() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9138) the name of function getHaseIntegrationTestingUtility() is a misspelling
[
https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chendihao updated HBASE-9138:
-
Description:
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a misspelling.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
was:the function getHaseIntegrationTestingUtility() in ChaosMonkey.java
should be getHBaseIntegrationTestingUtility()
the name of function getHaseIntegrationTestingUtility() is a misspelling
Key: HBASE-9138
URL: https://issues.apache.org/jira/browse/HBASE-9138
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.8
Reporter: chendihao
Priority: Trivial
Attachments: ChaosMonkey.java.patch
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a misspelling.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9138) the name of function getHaseIntegrationTestingUtility() is a misspelling
[
https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chendihao updated HBASE-9138:
-
Description:
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a spelling mistake.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
was:
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a misspelling.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
the name of function getHaseIntegrationTestingUtility() is a misspelling
Key: HBASE-9138
URL: https://issues.apache.org/jira/browse/HBASE-9138
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.8
Reporter: chendihao
Priority: Trivial
Attachments: ChaosMonkey.java.patch
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a spelling mistake.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9138) the name of function getHaseIntegrationTestingUtility() is a misspelling
[
https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chendihao updated HBASE-9138:
-
Affects Version/s: (was: 0.94.8)
0.94.4
the name of function getHaseIntegrationTestingUtility() is a misspelling
Key: HBASE-9138
URL: https://issues.apache.org/jira/browse/HBASE-9138
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.4
Reporter: chendihao
Priority: Trivial
Attachments: ChaosMonkey.java.patch
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a spelling mistake.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8332) Add truncate as HMaster method
[ https://issues.apache.org/jira/browse/HBASE-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-8332: --- Attachment: HBASE-8332.draft.patch Add truncate as HMaster method -- Key: HBASE-8332 URL: https://issues.apache.org/jira/browse/HBASE-8332 Project: HBase Issue Type: Improvement Components: master Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Attachments: HBASE-8332.draft.patch Currently truncate and truncate_preserve are only shell functions, and implemented as deleteTable() + createTable(). Using ACLs the user running truncate, must have rights to create a table and only global granted users can create tables. Add truncate() and truncatePreserve() to HBaseAdmin/HMaster with its own ACL check. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730755#comment-13730755 ] stack commented on HBASE-8741: -- I Region redundant in RegionSequenceIds since SequenceIds are per Region always when this patch applied. Mutations on Regions in recovery mode might have same sequenceIDs - Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.2 Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9136) RPC side changes to have a different codec for server to client communication
[ https://issues.apache.org/jira/browse/HBASE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730758#comment-13730758 ] stack commented on HBASE-9136: -- Replication is a different context again? We shuttle the KVs across using a different API. Could Codec figure this out? (Just a suggestion) RPC side changes to have a different codec for server to client communication - Key: HBASE-9136 URL: https://issues.apache.org/jira/browse/HBASE-9136 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 With reference to the mail sent in the dev list, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 We should have a provision such that the codec on the server side could be different from the one on the client side. This would help to remove the tags for security usecases. This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8741) Mutations on Regions in recovery mode might have same sequenceIDs
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730755#comment-13730755 ] stack edited comment on HBASE-8741 at 8/6/13 1:28 PM: -- Region is redundant in RegionSequenceIds since SequenceIds are per Region always when this patch applied. was (Author: stack): I Region redundant in RegionSequenceIds since SequenceIds are per Region always when this patch applied. Mutations on Regions in recovery mode might have same sequenceIDs - Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.2 Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] samar updated HBASE-9113: - Attachment: Screen Shot-table-details-V1.png Attaching a screen shot of how it is shaping up. Please validate if this make sense so that I can proceed with the development Expose region statistics on table.jsp - Key: HBASE-9113 URL: https://issues.apache.org/jira/browse/HBASE-9113 Project: HBase Issue Type: New Feature Components: Admin, UI Reporter: Bryan Beaudreault Assignee: samar Priority: Minor Attachments: Screen Shot-table-details-V1.png While Hannibal (https://github.com/sentric/hannibal) is great, the goal should be to eventually make it obsolete by providing the same features in the main HBase web UI (and HBaseAdmin API). The first step for that is region statistics on the table.jsp. Please provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730793#comment-13730793 ] Ted Yu commented on HBASE-9113: --- lgtm Expose region statistics on table.jsp - Key: HBASE-9113 URL: https://issues.apache.org/jira/browse/HBASE-9113 Project: HBase Issue Type: New Feature Components: Admin, UI Reporter: Bryan Beaudreault Assignee: samar Priority: Minor Attachments: Screen Shot-table-details-V1.png While Hannibal (https://github.com/sentric/hannibal) is great, the goal should be to eventually make it obsolete by providing the same features in the main HBase web UI (and HBaseAdmin API). The first step for that is region statistics on the table.jsp. Please provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730794#comment-13730794 ] Jean-Marc Spaggiari commented on HBASE-9113: Looks good to me too. Have you tried with huge regions names? Expose region statistics on table.jsp - Key: HBASE-9113 URL: https://issues.apache.org/jira/browse/HBASE-9113 Project: HBase Issue Type: New Feature Components: Admin, UI Reporter: Bryan Beaudreault Assignee: samar Priority: Minor Attachments: Screen Shot-table-details-V1.png While Hannibal (https://github.com/sentric/hannibal) is great, the goal should be to eventually make it obsolete by providing the same features in the main HBase web UI (and HBaseAdmin API). The first step for that is region statistics on the table.jsp. Please provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730795#comment-13730795 ] Bryan Beaudreault commented on HBASE-9113: -- +1 Expose region statistics on table.jsp - Key: HBASE-9113 URL: https://issues.apache.org/jira/browse/HBASE-9113 Project: HBase Issue Type: New Feature Components: Admin, UI Reporter: Bryan Beaudreault Assignee: samar Priority: Minor Attachments: Screen Shot-table-details-V1.png While Hannibal (https://github.com/sentric/hannibal) is great, the goal should be to eventually make it obsolete by providing the same features in the main HBase web UI (and HBaseAdmin API). The first step for that is region statistics on the table.jsp. Please provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Attachment: (was: HBASE-9126_V2.patch) Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Status: Open (was: Patch Available) Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730820#comment-13730820 ] stack commented on HBASE-9113: -- lgtm Where is the info for the page coming from? From ClusterStatus? Expose region statistics on table.jsp - Key: HBASE-9113 URL: https://issues.apache.org/jira/browse/HBASE-9113 Project: HBase Issue Type: New Feature Components: Admin, UI Reporter: Bryan Beaudreault Assignee: samar Priority: Minor Attachments: Screen Shot-table-details-V1.png While Hannibal (https://github.com/sentric/hannibal) is great, the goal should be to eventually make it obsolete by providing the same features in the main HBase web UI (and HBaseAdmin API). The first step for that is region statistics on the table.jsp. Please provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Attachment: HBASE-9126_V2.patch Reattaching to make the QA run Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch, HBASE-9126_V2.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Status: Patch Available (was: Open) Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch, HBASE-9126_V2.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9138) the name of function getHaseIntegrationTestingUtility() is a misspelling
[
https://issues.apache.org/jira/browse/HBASE-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730821#comment-13730821
]
stack commented on HBASE-9138:
--
Can you take the diff at the top-level of your hbase install; i.e. cd
HBASE_HOMEDIR; svn diff . (or git diff)? Thanks for fixing this.
the name of function getHaseIntegrationTestingUtility() is a misspelling
Key: HBASE-9138
URL: https://issues.apache.org/jira/browse/HBASE-9138
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.4
Reporter: chendihao
Priority: Trivial
Attachments: ChaosMonkey.java.patch
The function getHaseIntegrationTestingUtility() in ChaosMonkey.java should be
getHBaseIntegrationTestingUtility(), just a spelling mistake.
{code}
/**
* Context for Action's
*/
public static class ActionContext {
private IntegrationTestingUtility util;
public ActionContext(IntegrationTestingUtility util) {
this.util = util;
}
public IntegrationTestingUtility getHaseIntegrationTestingUtility() {
return util;
}
public HBaseCluster getHBaseCluster() {
return util.getHBaseClusterInterface();
}
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9113) Expose region statistics on table.jsp
[ https://issues.apache.org/jira/browse/HBASE-9113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730824#comment-13730824 ] Bryan Beaudreault commented on HBASE-9113: -- If new code was required to pull this info together in this view, it'd be awesome to expose in ClusterStatus or another HBaseAdmin API. Expose region statistics on table.jsp - Key: HBASE-9113 URL: https://issues.apache.org/jira/browse/HBASE-9113 Project: HBase Issue Type: New Feature Components: Admin, UI Reporter: Bryan Beaudreault Assignee: samar Priority: Minor Attachments: Screen Shot-table-details-V1.png While Hannibal (https://github.com/sentric/hannibal) is great, the goal should be to eventually make it obsolete by providing the same features in the main HBase web UI (and HBaseAdmin API). The first step for that is region statistics on the table.jsp. Please provide the same statistics per-region on table.jsp as in rs-status.jsp. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8778:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk and 0.95 (ns needs another revision anyways). Thanks D.
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
{noformat}
To open the region, the region server first loads the latest
HTableDescriptor. Since HBASE-4553 HTableDescriptor's are stored in the file
system at /hbase/tableDir/.tableinfo.sequenceNum. The file with the
largest sequenceNum is the current descriptor. This is done so that the
current descirptor is updated atomically. However, since the filename is not
known in advance FSTableDescriptors it has to do a FileSystem.listStatus
operation which has to list all files in the directory to find it. The
directory also contains all the region directories, so in our case it has to
load 130k FileStatus objects. Even using a globStatus matching function
still transfers all the objects to the client before performing the pattern
matching. Furthermore HDFS uses a default of transferring 1000 directory
entries in each RPC call, so it requires 130 roundtrips to the namenode to
fetch all the directory entries.
Consequently, to reassign all the regions of a table (or a constant fraction
thereof) requires time proportional to the square of the number of regions.
In our case, if a region server fails with 200 such regions, it takes 10+
minutes for them all to be reassigned, after the zk expiration and log
splitting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA
[jira] [Commented] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730835#comment-13730835
]
stack commented on HBASE-8778:
--
[~davelatham] Mind adding a release note? Just high level on what gets moved
where? Thanks boss...
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
{noformat}
To open the region, the region server first loads the latest
HTableDescriptor. Since HBASE-4553 HTableDescriptor's are stored in the file
system at /hbase/tableDir/.tableinfo.sequenceNum. The file with the
largest sequenceNum is the current descriptor. This is done so that the
current descirptor is updated atomically. However, since the filename is not
known in advance FSTableDescriptors it has to do a FileSystem.listStatus
operation which has to list all files in the directory to find it. The
directory also contains all the region directories, so in our case it has to
load 130k FileStatus objects. Even using a globStatus matching function
still transfers all the objects to the client before performing the pattern
matching. Furthermore HDFS uses a default of transferring 1000 directory
entries in each RPC call, so it requires 130 roundtrips to the namenode to
fetch all the directory entries.
Consequently, to reassign all the regions of a table (or a constant fraction
thereof) requires time proportional to the square of the number of regions.
In our case, if a region server fails with 200 such regions, it takes 10+
minutes for them all to be reassigned, after the zk expiration and log
splitting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For
[jira] [Commented] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730856#comment-13730856
]
Hudson commented on HBASE-8778:
---
FAILURE: Integrated in hbase-0.95 #407 (See
[https://builds.apache.org/job/hbase-0.95/407/])
HBASE-8778 Region assigments scan table directory making them slow for huge
tables (stack: rev 1510978)
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptorMigrationToSubdir.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDescriptorModification.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at
[jira] [Updated] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8778:
---
Release Note: Table descriptors are now moved inside hdfs from residing
directly in the table directory (along side region directories) to being in a
well known subdirectory called .tabledesc. For example, instead of
/hbase/exampleTable/.tableinfo.03 the file would be
/hbase/exampleTable/.tabledesc/.tableinfo.03 after this release. The
same will be true for snapshots. The first active master to be started up will
move these files for existing tables and snapshots.
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
{noformat}
To open the region, the region server first loads the latest
HTableDescriptor. Since HBASE-4553 HTableDescriptor's are stored in the file
system at /hbase/tableDir/.tableinfo.sequenceNum. The file with the
largest sequenceNum is the current descriptor. This is done so that the
current descirptor is updated atomically. However, since the filename is not
known in advance FSTableDescriptors it has to do a FileSystem.listStatus
operation which has to list all files in the directory to find it. The
directory also contains all the region directories, so in our case it has to
load 130k FileStatus objects. Even using a globStatus matching function
still transfers all the objects to the client before performing the pattern
matching. Furthermore HDFS uses a default of transferring 1000 directory
entries in each RPC call, so it requires 130 roundtrips to the namenode to
fetch all the directory entries.
Consequently, to reassign all the regions of a table (or a constant fraction
thereof) requires time proportional
[jira] [Updated] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Latham updated HBASE-8778:
---
Release Note: Table descriptors are now moved inside hdfs from residing
directly in the table directory (alongside region directories) to being in a
well known subdirectory called .tabledesc. For example, instead of
/hbase/exampleTable/.tableinfo.03 the file would be
/hbase/exampleTable/.tabledesc/.tableinfo.03 after this release. The
same will be true for snapshots. The first active master to be started up will
move these files for existing tables and snapshots. (was: Table descriptors
are now moved inside hdfs from residing directly in the table directory (along
side region directories) to being in a well known subdirectory called
.tabledesc. For example, instead of
/hbase/exampleTable/.tableinfo.03 the file would be
/hbase/exampleTable/.tabledesc/.tableinfo.03 after this release. The
same will be true for snapshots. The first active master to be started up will
move these files for existing tables and snapshots.)
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:255)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoModtime(FSTableDescriptors.java:368)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:155)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:126)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2834)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2807)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
{noformat}
To open the region, the region server first loads the latest
HTableDescriptor. Since HBASE-4553 HTableDescriptor's are stored in the file
system at /hbase/tableDir/.tableinfo.sequenceNum. The file with the
largest sequenceNum is the current descriptor. This is done so that the
current descirptor is updated atomically. However, since the filename is not
known in advance FSTableDescriptors it has to do a FileSystem.listStatus
operation which has to list all files in the directory to find it. The
directory also contains all the region
[jira] [Created] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
Julian Zhou created HBASE-9139: -- Summary: Independent timeout configuration for rpc channel between cluster nodes Key: HBASE-9139 URL: https://issues.apache.org/jira/browse/HBASE-9139 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.94.10, 0.96.0 Reporter: Julian Zhou Priority: Minor Fix For: 0.94.11, 0.96.0 Default of hbase.rpc.timeout is 6 ms (1 min). User sometimes increase them to a bigger value such as 60 ms (10 mins) for many concurrent loading application from client. Some user share the same hbase-site.xml for both client and server. HRegionServer #tryRegionServerReport via rpc channel to report to live master, but there was a window for master failover scenario. That region server attempting to connect to master, which was just killed, backup master took the active role immediately and put to /hbase/master, but region server was still waiting for the rpc timeout from connecting to the dead master. If hbase.rpc.timeout is too long, this master failover process will be long due to long rpc timeout from dead master. If so, could we separate with 2 options, hbase.rpc.timeout is still for hbase client, while hbase.rpc.internal.timeout was for this regionserver/master rpc channel, which could be set shorted value without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9126) Make HFile MIN VERSION as 2
[
https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730860#comment-13730860
]
Hadoop QA commented on HBASE-9126:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596353/HBASE-9126_V2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6617//console
This message is automatically generated.
Make HFile MIN VERSION as 2
---
Key: HBASE-9126
URL: https://issues.apache.org/jira/browse/HBASE-9126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.95.1
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0, 0.95.2
Attachments: HBASE-9126.patch, HBASE-9126_V2.patch
Removed the HFile V1 support from version95. We can make the min version
supported as 2?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730870#comment-13730870
]
Hudson commented on HBASE-8778:
---
SUCCESS: Integrated in hbase-0.95-on-hadoop2 #221 (See
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/221/])
HBASE-8778 Region assigments scan table directory making them slow for huge
tables (stack: rev 1510978)
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptorMigrationToSubdir.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDescriptorModification.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730871#comment-13730871 ] Nicolas Liochon commented on HBASE-9139: Even a 1 minute timeout is not ideal in this case: we know that the work to do server side is trivial, and we know it's idempotent so we can retry. So I would to tend to use a specific setting to use for such operations. It would be case by case. I don't have a good name for this setting, may be something like hbase.rpc.short.operation.timeout Independent timeout configuration for rpc channel between cluster nodes --- Key: HBASE-9139 URL: https://issues.apache.org/jira/browse/HBASE-9139 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.94.10, 0.96.0 Reporter: Julian Zhou Priority: Minor Fix For: 0.94.11, 0.96.0 Default of hbase.rpc.timeout is 6 ms (1 min). User sometimes increase them to a bigger value such as 60 ms (10 mins) for many concurrent loading application from client. Some user share the same hbase-site.xml for both client and server. HRegionServer #tryRegionServerReport via rpc channel to report to live master, but there was a window for master failover scenario. That region server attempting to connect to master, which was just killed, backup master took the active role immediately and put to /hbase/master, but region server was still waiting for the rpc timeout from connecting to the dead master. If hbase.rpc.timeout is too long, this master failover process will be long due to long rpc timeout from dead master. If so, could we separate with 2 options, hbase.rpc.timeout is still for hbase client, while hbase.rpc.internal.timeout was for this regionserver/master rpc channel, which could be set shorted value without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9116) Add a view/edit tool for favored node mappings for regions
[
https://issues.apache.org/jira/browse/HBASE-9116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730882#comment-13730882
]
Ted Yu commented on HBASE-9116:
---
Useful tool :-)
{code}
+ListFloat dispersion = report.getDispersionInformation();
+if (simplePrint) {
+ DecimalFormat df = new java.text.DecimalFormat(#.##);
+ System.out.println(\tAvg dispersion score:
+ + df.format(dispersion.get(0)) + hosts;\tMax dispersion score:
{code}
It would be easier to maintain the code if index to dispersions has
corresponding enum.
For FSRegionScanner :
{code}
+ * Maps region encoded names to maps of hostnames to fractional locality of
+ * that region on that host.
+ */
+ private MapString, MapString, Float regionDegreeLocalityMapping;
{code}
Should ServerName be used as key for the inner Map above ?
{code}
+// not part of a table?
+if (parent.getName().startsWith(.)
+ !parent.getName().equals(.META.)) {
{code}
The above would need adjustment once namespace patch goes in.
RegionPlacementMaintainer.java needs license
{code}
+ * @param newPlanMap - new AssignmentPlan
+ * @return how many primaries will move per table
+ */
+ public MapString, Integer getRegionsMovement(FavoredNodes newPlan)
{code}
Parameter and its javadoc don't seem to match.
Add a view/edit tool for favored node mappings for regions
--
Key: HBASE-9116
URL: https://issues.apache.org/jira/browse/HBASE-9116
Project: HBase
Issue Type: Improvement
Components: Region Assignment
Affects Versions: 0.95.0
Reporter: Devaraj Das
Assignee: Devaraj Das
Attachments: 9116-1.txt
Add a tool that one can run offline to view the favored node mappings for
regions, and also fix the mappings if needed. Such a tool exists in the
0.89-fb branch. Will port it over to trunk/0.95.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7391) Review/improve HLog compression's memory consumption
[
https://issues.apache.org/jira/browse/HBASE-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-7391:
-
Assignee: ramkrishna.s.vasudevan
Review/improve HLog compression's memory consumption
Key: HBASE-7391
URL: https://issues.apache.org/jira/browse/HBASE-7391
Project: HBase
Issue Type: Bug
Reporter: Jean-Daniel Cryans
Assignee: ramkrishna.s.vasudevan
Fix For: 0.95.2
From Ram in
http://mail-archives.apache.org/mod_mbox/hbase-dev/201205.mbox/%3C00bc01cd31e6$7caf1320$760d3960$%25vasude...@huawei.com%3E:
{quote}
One small observation after giving +1 on the RC.
The WAL compression feature causes OOME and causes Full GC.
The problem is, if we have 1500 regions and I need to create recovered.edits
for each of the region (I don’t have much data in the regions (~300MB)).
Now when I try to build the dictionary there is a Node object getting
created.
Each node object occupies 32 bytes.
We have 5 such dictionaries.
Initially we create indexToNodes array and its size is 32767.
So now we have 32*5*32767 = ~5MB.
Now I have 1500 regions.
So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high
initial memory foot print and this never allows me to split the logs and I
am not able to make the cluster up at all.
Our configured heap size was 8GB, tested in 3 node cluster with 5000
regions, very less data( 1GB in hdfs cluster including replication), some
small data is spread evenly across all regions.
The formula is 32(Node object size)*5(No of dictionary)*32767(no of node
objects)*noofregions.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9131) Add admin-level documention about configuration and usage of the Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-9131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730883#comment-13730883
]
Jonathan Hsieh commented on HBASE-9131:
---
[~zjushch] Thanks. I think we are somewhere between too little detail and too
much detail.
First, can we add the config variables to hbase-default.xml (with full
descriptions and with units).
Now to the meat:
The patch doesn't tell the admin why or when they'd want to consider using
this. The link/pdf requires having to search for the bucket cache sections in
the 2nd page and then goes on into too much design detail for an average admin.
(It also lacks the config variables / instructions).
My suggestion: Take let's take the high-level parts from section 3 of the pdf,
polish it and add it to the official docs.
Here's a stab at the sections that I think would be good for the ref guide with
the prose improved a little bit:
{quote}
*Design and Motivation*
The Bucket Cache is an alternate block cache implementation that is designed to
take advantage of large amounts of memory or low-latency storage. (something
about how big would be useful). It is implemented as an off-the-jvm-heap and
which has the secondary benefit of reducing JVM heap fragmentation that
eventually causes stop-the-world JVM garbage collection operations. If one were
to rely upon the standard JVM memory allocation and GC policies with large
heaps (16GB RAM) one would periodically incur instability in hbase due to long
stop-the-world GC pauses (10's of secs to minutes) that can be misinterpreted
as region server failures.
The storage of cached blocks is is not constrained to in RAM-only use; one
could cache blocks in memory and also use a high speed disk, such as SSD's,
Fusion-IO devices, or ram-disks as massive secondary cache. (probably need
something about the persistence properties not being required, but having the
masssive capacity as a huge benefit.
Internally, the bucket cache divided storage into many *buckets*, each of which
contains blocks of a particular range of sizes. (this is a little fuzzy, needs
some clarification). Insertions and evictions of blocks backed by physical
storage just overwrites blocks on the device or reads data from the storage
device. Managing these larger blocks prevents external fragmentation that
causes GC pauses at the cost of some minor wasted space (internal
fragmentation).
*Configuration and Usage*
To configure the bucket cache... (something along the line of what the current
patch has)
{quote}
Let me know what you think, and feel free to update/correct the draft.
Add admin-level documention about configuration and usage of the Bucket Cache
-
Key: HBASE-9131
URL: https://issues.apache.org/jira/browse/HBASE-9131
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Attachments: hbase-9131.patch
HBASE-7404 added the bucket cache but its configuration settings are
currently undocumented. Without documentation developers would be the only
ones aware of the feature.
Specifically documentation about slide 23 from
http://www.slideshare.net/cloudera/operations-session-4 would be great to add!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730889#comment-13730889 ] Nick Dimiduk commented on HBASE-9139: - Does it make more sense from a user perspective to have a global system setting called hbase.rpc.timeout and then allow for it to be overridden at the client level with hbase.client.rpc.timeout? This allows us to introduce alternative configurations for other timeout properties within the internal category, if that becomes necessary. (It also keeps the name consistent with other client-oriented configuration parameters.) Independent timeout configuration for rpc channel between cluster nodes --- Key: HBASE-9139 URL: https://issues.apache.org/jira/browse/HBASE-9139 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.94.10, 0.96.0 Reporter: Julian Zhou Priority: Minor Fix For: 0.94.11, 0.96.0 Default of hbase.rpc.timeout is 6 ms (1 min). User sometimes increase them to a bigger value such as 60 ms (10 mins) for many concurrent loading application from client. Some user share the same hbase-site.xml for both client and server. HRegionServer #tryRegionServerReport via rpc channel to report to live master, but there was a window for master failover scenario. That region server attempting to connect to master, which was just killed, backup master took the active role immediately and put to /hbase/master, but region server was still waiting for the rpc timeout from connecting to the dead master. If hbase.rpc.timeout is too long, this master failover process will be long due to long rpc timeout from dead master. If so, could we separate with 2 options, hbase.rpc.timeout is still for hbase client, while hbase.rpc.internal.timeout was for this regionserver/master rpc channel, which could be set shorted value without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-9140) NPE in AsyncProcess
Elliott Clark created HBASE-9140:
Summary: NPE in AsyncProcess
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9135) Upgrade hadoop 1 version to 1.2.1 which is stable
[ https://issues.apache.org/jira/browse/HBASE-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730893#comment-13730893 ] Anoop Sam John commented on HBASE-9135: --- LGTM. Upgrade hadoop 1 version to 1.2.1 which is stable - Key: HBASE-9135 URL: https://issues.apache.org/jira/browse/HBASE-9135 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Richard Ding Attachments: 9135-v1.txt Here is related discussion: http://search-hadoop.com/m/nA71y1kKHDm1/Hadoop+version+1.2.1+%2528stable%2529+releasedsubj=Re+ANNOUNCE+Hadoop+version+1+2+1+stable+released Older hadoop 1 artifacts would be phased out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[
https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730894#comment-13730894
]
Nick Dimiduk commented on HBASE-9091:
-
Make {{ByteRange}} an abstract class and you can have a single implementation
of most of the logic. Does that lose for you the desired level of abstraction?
Maybe a ByteRangeComparator can hold the comparison logic?
How important is having this be an interface, considering it's client-direct
API? Are we holding out on this level of abstraction?
Update ByteRange to maintain consumer's position
Key: HBASE-9091
URL: https://issues.apache.org/jira/browse/HBASE-9091
Project: HBase
Issue Type: Improvement
Components: Client
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Attachments: 0001-HBASE-9091-Extend-ByteRange.patch,
0001-HBASE-9091-Extend-ByteRange.patch,
0001-HBASE-9091-Introduce-PositionedByteRange.patch,
0001-HBASE-9091-Introduce-PositionedByteRange.patch,
0001-HBASE-9091-Introduce-PositionedByteRange.patch
ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is
mutable and an instance can be assigned over a byte[] after instantiation.
This is valuable as a performance consideration when working with byte[]
slices in a tight loop. Its current design is such that it is not possible to
consume a portion of the range while performing activities like decoding an
object without altering the definition of the range. It should provide a
position that is independent from the range's offset and length to make
partial reads easier.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730898#comment-13730898 ] Nicolas Liochon commented on HBASE-9139: I think it could become difficult to manage: them we would have half-internal client like map reduce and so one. So to me, if you want different settings just use different configuration files. The real differences is in the operations imho. Some operations take no time and can be retried. So having the same timeout for such operations and for operations that can take longer (for example because they write in hdfs) is not good. Internally, in HBase code, we should use a different setting for such operations. Independent timeout configuration for rpc channel between cluster nodes --- Key: HBASE-9139 URL: https://issues.apache.org/jira/browse/HBASE-9139 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.94.10, 0.96.0 Reporter: Julian Zhou Priority: Minor Fix For: 0.94.11, 0.96.0 Default of hbase.rpc.timeout is 6 ms (1 min). User sometimes increase them to a bigger value such as 60 ms (10 mins) for many concurrent loading application from client. Some user share the same hbase-site.xml for both client and server. HRegionServer #tryRegionServerReport via rpc channel to report to live master, but there was a window for master failover scenario. That region server attempting to connect to master, which was just killed, backup master took the active role immediately and put to /hbase/master, but region server was still waiting for the rpc timeout from connecting to the dead master. If hbase.rpc.timeout is too long, this master failover process will be long due to long rpc timeout from dead master. If so, could we separate with 2 options, hbase.rpc.timeout is still for hbase client, while hbase.rpc.internal.timeout was for this regionserver/master rpc channel, which could be set shorted value without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730897#comment-13730897 ] Ted Yu commented on HBASE-9126: --- +1 Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch, HBASE-9126_V2.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9132) Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94
[ https://issues.apache.org/jira/browse/HBASE-9132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9132: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.94. Thanks for the patch, Dave. Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94 -- Key: HBASE-9132 URL: https://issues.apache.org/jira/browse/HBASE-9132 Project: HBase Issue Type: Improvement Reporter: Dave Latham Assignee: Dave Latham Fix For: 0.94.11 Attachments: HBASE-9132-0.94.patch As described in HBASE-8778, anything that needs to read table descriptors (such as region assignments) currently results in a table directory scan which is slow on tables with many regions. HBASE-8778 moved the table descriptors to a subdir for 0.96+. For 0.94 a smaller change that is safe with rolling restarts is to cache the directory modtime and use the cached table descriptor if the directory modtime has not changed since a table descriptor change always changes the directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9139: - Fix Version/s: (was: 0.94.11) 0.94.12 Independent timeout configuration for rpc channel between cluster nodes --- Key: HBASE-9139 URL: https://issues.apache.org/jira/browse/HBASE-9139 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.94.10, 0.96.0 Reporter: Julian Zhou Priority: Minor Fix For: 0.94.12, 0.96.0 Default of hbase.rpc.timeout is 6 ms (1 min). User sometimes increase them to a bigger value such as 60 ms (10 mins) for many concurrent loading application from client. Some user share the same hbase-site.xml for both client and server. HRegionServer #tryRegionServerReport via rpc channel to report to live master, but there was a window for master failover scenario. That region server attempting to connect to master, which was just killed, backup master took the active role immediately and put to /hbase/master, but region server was still waiting for the rpc timeout from connecting to the dead master. If hbase.rpc.timeout is too long, this master failover process will be long due to long rpc timeout from dead master. If so, could we separate with 2 options, hbase.rpc.timeout is still for hbase client, while hbase.rpc.internal.timeout was for this regionserver/master rpc channel, which could be set shorted value without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9139) Independent timeout configuration for rpc channel between cluster nodes
[ https://issues.apache.org/jira/browse/HBASE-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730905#comment-13730905 ] Lars Hofhansl commented on HBASE-9139: -- I agree with Nicolas. The operation that Julian mentions definitely falls into that category. Independent timeout configuration for rpc channel between cluster nodes --- Key: HBASE-9139 URL: https://issues.apache.org/jira/browse/HBASE-9139 Project: HBase Issue Type: Improvement Components: IPC/RPC, regionserver Affects Versions: 0.94.10, 0.96.0 Reporter: Julian Zhou Priority: Minor Fix For: 0.94.11, 0.96.0 Default of hbase.rpc.timeout is 6 ms (1 min). User sometimes increase them to a bigger value such as 60 ms (10 mins) for many concurrent loading application from client. Some user share the same hbase-site.xml for both client and server. HRegionServer #tryRegionServerReport via rpc channel to report to live master, but there was a window for master failover scenario. That region server attempting to connect to master, which was just killed, backup master took the active role immediately and put to /hbase/master, but region server was still waiting for the rpc timeout from connecting to the dead master. If hbase.rpc.timeout is too long, this master failover process will be long due to long rpc timeout from dead master. If so, could we separate with 2 options, hbase.rpc.timeout is still for hbase client, while hbase.rpc.internal.timeout was for this regionserver/master rpc channel, which could be set shorted value without affect real client rpc timeout value? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9140) NPE in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730912#comment-13730912
]
Nicolas Liochon commented on HBASE-9140:
That's this line, yes?
loc = hConnection.locateRegion(this.tableName, row.getRow());
hConnection is final, so it should come from row...
NPE in AsyncProcess
---
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9126: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to Trunk and 0.95. Thanks for the review Stack Ted Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch, HBASE-9126_V2.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8627: -- Attachment: (was: HBASE-8627_Trunk-V2.patch) HBCK can not fix meta not assigned issue Key: HBASE-8627 URL: https://issues.apache.org/jira/browse/HBASE-8627 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.95.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-8627_Trunk.patch When meta table region is not assigned to any RS, HBCK run will get exception. I can see code added in checkMetaRegion() to solve this issue but it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8627: -- Status: Open (was: Patch Available) HBCK can not fix meta not assigned issue Key: HBASE-8627 URL: https://issues.apache.org/jira/browse/HBASE-8627 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.95.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-8627_Trunk.patch When meta table region is not assigned to any RS, HBCK run will get exception. I can see code added in checkMetaRegion() to solve this issue but it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8627: -- Attachment: HBASE-8627_Trunk-V2.patch Reattaching. Let QA run once again. As per the result I will commit HBCK can not fix meta not assigned issue Key: HBASE-8627 URL: https://issues.apache.org/jira/browse/HBASE-8627 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.95.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-8627_Trunk.patch, HBASE-8627_Trunk-V2.patch When meta table region is not assigned to any RS, HBCK run will get exception. I can see code added in checkMetaRegion() to solve this issue but it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8627: -- Status: Patch Available (was: Open) HBCK can not fix meta not assigned issue Key: HBASE-8627 URL: https://issues.apache.org/jira/browse/HBASE-8627 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.95.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-8627_Trunk.patch, HBASE-8627_Trunk-V2.patch When meta table region is not assigned to any RS, HBCK run will get exception. I can see code added in checkMetaRegion() to solve this issue but it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9140) NPE in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730930#comment-13730930
]
Nicolas Liochon commented on HBASE-9140:
I looked at the code, I don't have a clear scenario. When we add to the list,
it can't be null, we have
currentWriteBufferSize += put.heapSize();
writeAsyncBuffer.add(put);
so if 'put' is null we NPE before adding it to the list.
You're working on it Elliott? What's the ycsb setting you used?
NPE in AsyncProcess
---
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9079: - Fix Version/s: 0.94.11 0.95.2 0.98.0 FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9140) NPE in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730939#comment-13730939
]
Elliott Clark commented on HBASE-9140:
--
HBase Git rev:
{code} f760bf88b67f609d3fcf18c317139361ff5ad1df{code}
Table setup:
{code}create 'usertable',{NAME = 'd', VERSIONS = 1,COMPRESSION = 'snappy',
BLOCKCACHE = true}{code}
Ycsb command:
{code}bin/ycsb load hbase -P workloads/workloada -p columnfamily=d -s -threads
32 -p recordcount=4000{code}
It was an automated YCSB run. I didn't notice it until this morning that the
run was less than half of the normal time. Then looking though the logs I
found the error. It hasn't happened since and I can't get it to reproduce, but
it's weird.
NPE in AsyncProcess
---
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[
https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730946#comment-13730946
]
Francis Liu commented on HBASE-8409:
{quote}
On rewriting .regioninfo, FYI HBASE-8778 moves stuff around and is about to go
in.
{quote}
Looks like it got in. Will have to rebase and start working on the upgrade.
Hopefully we can get namespace in soon.
Security support for namespaces
---
Key: HBASE-8409
URL: https://issues.apache.org/jira/browse/HBASE-8409
Project: HBase
Issue Type: Sub-task
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Priority: Critical
Attachments: HBASE-8049_trunk.patch
This task adds the security piece to the namespace feature. The work related
to migration of the existing acl table to the new namespace is remaining and
will be completed in the follow up patch. Permissions can be granted to a
namespace by the hbase admin, by appending '@' to the namespace name. A user
with write or admin permissions on a given namespace can create tables in
that namespace. The other privileges (R, X, C ) do not have any special
meaning w.r.t namespaces. Any users of hbase can list tables in a namespace.
The following commands can only be executed by HBase admins.
1. Grant privileges for user on Namespace.
2. Revoke privileges for user on Namespace
Grant Command:
hbase grant 'tenant-A' 'W' '@N1'
In the above example, the command will grant the user 'tenant-A' write
privileges for a namespace named N1.
Revoke Command:
hbase revoke 'tenant-A''@N1'
In the above example, the command will revoke all privileges from user
'tenant-A' for namespace named N1.
Lets see an example on how privileges work with namespaces.
User Mike request for a namespace named hbase_perf with the hbase admin.
whoami: hbase
hbase shell namespace_create 'hbase_perf'
hbase shell grant 'mike', 'W', '@hbase_perf'
Mike creates two tables table20 and table50 in the above workspace.
whoami: mike
hbase shell create 'hbase_perf.table20', 'family1'
hbase shell create 'hbase_perf.table50', 'family1'
Note: As Mike was able to create tables 'hbase_perf.table20',
'hbase_perf.table50', he becomes the owner of those tables.
This means he has RWXCA perms on those tables.
Another team member of Mike, Alice wants also to share the same workspace
hbase_perf. HBase admin grants Alice also permission to create tables in
hbase_perf namespace.
whoami: hbase
hbase shell grant 'alice', 'W', '@hbase_perf'
Now Alice can create new tables under hbase_perf namespace, but cannot
read,write,alter,delete existing tables in the namespace.
whoami: alice
hbase shell namespace_list_tables 'hbase_perf'
hbase_perf.table20
hbase_perf.table50
hbase shell scan 'hbase_perf.table20'
AccessDeniedException
If Alice wants to read or write to existing tables in the hbase_perf
namespace, hbase admins need to explicitly grant permission.
whoami: hbase
hbase shell grant 'alice', 'RW', 'hbase_perf.table20'
hbase shell grant 'alice', 'RW', 'hbase_perf.table50'
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9129) Scanner prefetching leaks scanners.
[
https://issues.apache.org/jira/browse/HBASE-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730947#comment-13730947
]
Elliott Clark commented on HBASE-9129:
--
So this did help my ycsb runs. However there is still some pretty huge
contention in Leases.addLease.
Scanner prefetching leaks scanners.
---
Key: HBASE-9129
URL: https://issues.apache.org/jira/browse/HBASE-9129
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.2
Attachments: HBASE-9129-0.patch
Running YCSB workload-e on 0.95 basically hangs the whole cluster.
YCSB opens a scanner for 100 rows.
# YCSB Calls next
# Pre-fetching starts
# YCSB Closes scanner
# Pre-fetching re-adds the scanner.
So the end result is:
{noformat}
scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc
runnable [0x7f75ffefd000]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272)
at java.util.PriorityQueue.indexOf(PriorityQueue.java:342)
at java.util.PriorityQueue.remove(PriorityQueue.java:360)
at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476)
at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232)
- locked 0x7f774455a660 (a java.util.concurrent.DelayQueue)
at
org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269)
at
org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb
waiting on condition [0x7f7605b79000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x7f774455ad78 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba
waiting on condition [0x7f7609ab8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x7f774455ad78 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722){noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9140) NPE in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730965#comment-13730965
]
Nicolas Liochon commented on HBASE-9140:
I don't even see where it would be useful to add logs to help debugging. This
code is called only by HTable, and HTable is not supposed to be used by
multiple threads.
May be the issue comes from here:
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
I mean, the YCSB code doing the cleanup could be executed in parallel to the
'main' one, or is passing a HTable object from a thread to another, with some
nasty processor cache effects?
NPE in AsyncProcess
---
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730975#comment-13730975 ] Andrew Purtell commented on HBASE-8409: --- bq. Namespace permissions should not be based on privileges in acl table, but rather global perms. It's late. These are just thoughts. The ability to create or delete namespaces should require GLOBAL ADMIN. The ability to create tables within a namespace should require ADMIN or CREATE grants on the namespace. (Isn't it confusing to be able to create tables if you have WRITE access on a namespace but not CREATE? CREATE has no meaning?) Any user permissions granted on a namespace authorize to all tables in the namespace. Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Priority: Critical Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9091) Update ByteRange to maintain consumer's position
[ https://issues.apache.org/jira/browse/HBASE-9091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730974#comment-13730974 ] Matt Corgan commented on HBASE-9091: I was thinking the interface would be necessary so that we can have different atomicity guarantees on different implementations' fields. Like the volatiles mentioned above. Or maybe one implementation just wraps a DirectByteBuffer in which case that's the only field. Update ByteRange to maintain consumer's position Key: HBASE-9091 URL: https://issues.apache.org/jira/browse/HBASE-9091 Project: HBase Issue Type: Improvement Components: Client Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Extend-ByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch, 0001-HBASE-9091-Introduce-PositionedByteRange.patch ByteRange is a useful alternative to Java's ByteBuffer. Notably, it is mutable and an instance can be assigned over a byte[] after instantiation. This is valuable as a performance consideration when working with byte[] slices in a tight loop. Its current design is such that it is not possible to consume a portion of the range while performing activities like decoding an object without altering the definition of the range. It should provide a position that is independent from the range's offset and length to make partial reads easier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8849) CellCodec should write and read the memstoreTS/mvccVersion
[ https://issues.apache.org/jira/browse/HBASE-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-8849: -- Affects Version/s: (was: 0.95.1) CellCodec should write and read the memstoreTS/mvccVersion -- Key: HBASE-8849 URL: https://issues.apache.org/jira/browse/HBASE-8849 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: keyvalue Fix For: 0.98.0 Attachments: HBASE-8849_1.patch, HBASE-8849_2.patch, HBASE-8849_3.patch This JIRA is wrt discussion over in HBASE-8496. Cell interface and the corresponding CellCodec provides a new way of serializing the Keyvalues. Cell interface supports getMvccVersion() and the javadoc says it could be 0 if it exists or 0 otherwise. But we don't tend to write/read the memstoreTS/mvccVersion in the Cell codec. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8015) Support for Namespaces
[ https://issues.apache.org/jira/browse/HBASE-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730979#comment-13730979 ] stack commented on HBASE-8015: -- [~enis] Where you at on your review? You want an update of the patch by [~toffer]? I committed HBASE-8778 because it looked like we needed a new revision. If current patch is good by you [~enis], I could back out HBASE-8778 because it would be easier applying that on top of ns. Support for Namespaces -- Key: HBASE-8015 URL: https://issues.apache.org/jira/browse/HBASE-8015 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Francis Liu Attachments: HBASE-8015_draft_94.patch, Namespace Design.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8849) CellCodec should write and read the memstoreTS/mvccVersion
[ https://issues.apache.org/jira/browse/HBASE-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-8849: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk i.e. 0.98. CellCodec should write and read the memstoreTS/mvccVersion -- Key: HBASE-8849 URL: https://issues.apache.org/jira/browse/HBASE-8849 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: keyvalue Fix For: 0.98.0 Attachments: HBASE-8849_1.patch, HBASE-8849_2.patch, HBASE-8849_3.patch This JIRA is wrt discussion over in HBASE-8496. Cell interface and the corresponding CellCodec provides a new way of serializing the Keyvalues. Cell interface supports getMvccVersion() and the javadoc says it could be 0 if it exists or 0 otherwise. But we don't tend to write/read the memstoreTS/mvccVersion in the Cell codec. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8849) CellCodec should write and read the memstoreTS/mvccVersion
[ https://issues.apache.org/jira/browse/HBASE-8849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730981#comment-13730981 ] ramkrishna.s.vasudevan commented on HBASE-8849: --- Thanks for the review Stack. CellCodec should write and read the memstoreTS/mvccVersion -- Key: HBASE-8849 URL: https://issues.apache.org/jira/browse/HBASE-8849 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: keyvalue Fix For: 0.98.0 Attachments: HBASE-8849_1.patch, HBASE-8849_2.patch, HBASE-8849_3.patch This JIRA is wrt discussion over in HBASE-8496. Cell interface and the corresponding CellCodec provides a new way of serializing the Keyvalues. Cell interface supports getMvccVersion() and the javadoc says it could be 0 if it exists or 0 otherwise. But we don't tend to write/read the memstoreTS/mvccVersion in the Cell codec. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8409) Security support for namespaces
[ https://issues.apache.org/jira/browse/HBASE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730991#comment-13730991 ] Enis Soztutar commented on HBASE-8409: -- One more comment is that if the user has already existing permissions in the acl table, the migration script should rename that table, or copy the acl definitions. Shall we do it in another subtask? Security support for namespaces --- Key: HBASE-8409 URL: https://issues.apache.org/jira/browse/HBASE-8409 Project: HBase Issue Type: Sub-task Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Priority: Critical Attachments: HBASE-8049_trunk.patch This task adds the security piece to the namespace feature. The work related to migration of the existing acl table to the new namespace is remaining and will be completed in the follow up patch. Permissions can be granted to a namespace by the hbase admin, by appending '@' to the namespace name. A user with write or admin permissions on a given namespace can create tables in that namespace. The other privileges (R, X, C ) do not have any special meaning w.r.t namespaces. Any users of hbase can list tables in a namespace. The following commands can only be executed by HBase admins. 1. Grant privileges for user on Namespace. 2. Revoke privileges for user on Namespace Grant Command: hbase grant 'tenant-A' 'W' '@N1' In the above example, the command will grant the user 'tenant-A' write privileges for a namespace named N1. Revoke Command: hbase revoke 'tenant-A''@N1' In the above example, the command will revoke all privileges from user 'tenant-A' for namespace named N1. Lets see an example on how privileges work with namespaces. User Mike request for a namespace named hbase_perf with the hbase admin. whoami: hbase hbase shell namespace_create 'hbase_perf' hbase shell grant 'mike', 'W', '@hbase_perf' Mike creates two tables table20 and table50 in the above workspace. whoami: mike hbase shell create 'hbase_perf.table20', 'family1' hbase shell create 'hbase_perf.table50', 'family1' Note: As Mike was able to create tables 'hbase_perf.table20', 'hbase_perf.table50', he becomes the owner of those tables. This means he has RWXCA perms on those tables. Another team member of Mike, Alice wants also to share the same workspace hbase_perf. HBase admin grants Alice also permission to create tables in hbase_perf namespace. whoami: hbase hbase shell grant 'alice', 'W', '@hbase_perf' Now Alice can create new tables under hbase_perf namespace, but cannot read,write,alter,delete existing tables in the namespace. whoami: alice hbase shell namespace_list_tables 'hbase_perf' hbase_perf.table20 hbase_perf.table50 hbase shell scan 'hbase_perf.table20' AccessDeniedException If Alice wants to read or write to existing tables in the hbase_perf namespace, hbase admins need to explicitly grant permission. whoami: hbase hbase shell grant 'alice', 'RW', 'hbase_perf.table20' hbase shell grant 'alice', 'RW', 'hbase_perf.table50' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8778) Region assigments scan table directory making them slow for huge tables
[
https://issues.apache.org/jira/browse/HBASE-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731005#comment-13731005
]
Hudson commented on HBASE-8778:
---
SUCCESS: Integrated in HBase-TRUNK #4347 (See
[https://builds.apache.org/job/HBase-TRUNK/4347/])
HBASE-8778 Region assigments scan table directory making them slow for huge
tables (stack: rev 1510977)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptorMigrationToSubdir.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/handler/TestTableDescriptorModification.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSTableDescriptors.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTable.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
Region assigments scan table directory making them slow for huge tables
---
Key: HBASE-8778
URL: https://issues.apache.org/jira/browse/HBASE-8778
Project: HBase
Issue Type: Improvement
Reporter: Dave Latham
Assignee: Dave Latham
Priority: Critical
Fix For: 0.98.0, 0.95.2
Attachments: 8778-dirmodtime.txt, HBASE-8778-0.94.5.patch,
HBASE-8778-0.94.5-v2.patch, HBASE-8778.patch, HBASE-8778-v2.patch,
HBASE-8778-v3.patch, HBASE-8778-v4.patch, HBASE-8778-v5.patch
On a table with 130k regions it takes about 3 seconds for a region server to
open a region once it has been assigned.
Watching the threads for a region server running 0.94.5 that is opening many
such regions shows the thread opening the reigon in code like this:
{noformat}
PRI IPC Server handler 4 on 60020 daemon prio=10 tid=0x2aaac07e9000
nid=0x6566 runnable [0x4c46d000]
java.lang.Thread.State: RUNNABLE
at java.lang.String.indexOf(String.java:1521)
at java.net.URI$Parser.scan(URI.java:2912)
at java.net.URI$Parser.parse(URI.java:3004)
at java.net.URI.init(URI.java:736)
at org.apache.hadoop.fs.Path.initialize(Path.java:145)
at org.apache.hadoop.fs.Path.init(Path.java:126)
at org.apache.hadoop.fs.Path.init(Path.java:50)
at
org.apache.hadoop.hdfs.protocol.HdfsFileStatus.getFullPath(HdfsFileStatus.java:215)
at
org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:252)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:311)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:159)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:842)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:867)
at org.apache.hadoop.hbase.util.FSUtils.listStatus(FSUtils.java:1168)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:269)
at
[jira] [Created] (HBASE-9141) Save replication znodes while migrating to 0.96
Himanshu Vashishtha created HBASE-9141: -- Summary: Save replication znodes while migrating to 0.96 Key: HBASE-9141 URL: https://issues.apache.org/jira/browse/HBASE-9141 Project: HBase Issue Type: Improvement Components: migration, Replication Affects Versions: 0.94.10 Reporter: Himanshu Vashishtha Fix For: 0.95.2 While migrating to 0.96, we recommend deleting old znodes so users not face issues like HBASE-7766, and let HBase create them out of box. Though HBase tends to store only ephemeral data in zookeeper, replication has a different approach. Almost all of its data (state, peer info, logs, etc) is present in zookeeper. We would like to preserve them in order to not do re-adding of peers, and ensuring complete replication after we have migrated to 0.96. This jira adds a tool to serialize/de-serialize replication znodes to the underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Summary: Replication Znodes Backup Tool (was: Save replication znodes while migrating to 0.96) Replication Znodes Backup Tool -- Key: HBASE-9141 URL: https://issues.apache.org/jira/browse/HBASE-9141 Project: HBase Issue Type: Improvement Components: migration, Replication Affects Versions: 0.94.10 Reporter: Himanshu Vashishtha Fix For: 0.95.2 While migrating to 0.96, we recommend deleting old znodes so users not face issues like HBASE-7766, and let HBase create them out of box. Though HBase tends to store only ephemeral data in zookeeper, replication has a different approach. Almost all of its data (state, peer info, logs, etc) is present in zookeeper. We would like to preserve them in order to not do re-adding of peers, and ensuring complete replication after we have migrated to 0.96. This jira adds a tool to serialize/de-serialize replication znodes to the underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8627) HBCK can not fix meta not assigned issue
[
https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731010#comment-13731010
]
Hadoop QA commented on HBASE-8627:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12596367/HBASE-8627_Trunk-V2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6618//console
This message is automatically generated.
HBCK can not fix meta not assigned issue
Key: HBASE-8627
URL: https://issues.apache.org/jira/browse/HBASE-8627
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.95.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-8627_Trunk.patch, HBASE-8627_Trunk-V2.patch
When meta table region is not assigned to any RS, HBCK run will get
exception. I can see code added in checkMetaRegion() to solve this issue but
it wont work. It still refers to ROOT region!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-9135) Upgrade hadoop 1 version to 1.2.1 which is stable
[ https://issues.apache.org/jira/browse/HBASE-9135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-9135: - Assignee: Ted Yu (was: Richard Ding) Upgrade hadoop 1 version to 1.2.1 which is stable - Key: HBASE-9135 URL: https://issues.apache.org/jira/browse/HBASE-9135 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Attachments: 9135-v1.txt Here is related discussion: http://search-hadoop.com/m/nA71y1kKHDm1/Hadoop+version+1.2.1+%2528stable%2529+releasedsubj=Re+ANNOUNCE+Hadoop+version+1+2+1+stable+released Older hadoop 1 artifacts would be phased out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9123) Filter protobuf generated code from long line warning
[ https://issues.apache.org/jira/browse/HBASE-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731033#comment-13731033 ] Hudson commented on HBASE-9123: --- SUCCESS: Integrated in HBase-0.94-security #247 (See [https://builds.apache.org/job/HBase-0.94-security/247/]) HBASE-9123 Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94 (Dave Latham) (larsh: rev 1511018) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Filter protobuf generated code from long line warning - Key: HBASE-9123 URL: https://issues.apache.org/jira/browse/HBASE-9123 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.98.0 Attachments: 9123.patch, 9123-v2.patch For big patch, such as the one for namespace, there would be many changes in the protobuf generated code. See example here: https://builds.apache.org/job/PreCommit-HBASE-Build/6569/console We should filter protobuf generated code from long line warning -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9136) RPC side changes to have a different codec for server to client communication
[ https://issues.apache.org/jira/browse/HBASE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731047#comment-13731047 ] ramkrishna.s.vasudevan commented on HBASE-9136: --- We shuttle the KVs across using a different API. Could Codec figure this out? Yes. May be like how WAL does by passing the Compression to the codec and checking it for null and calling different encoder and decoder? This would mean that we would get different implementation for the same codec? But in the RPC layer we would not be able to differentiate the options. Let me think through on this more and comment here. RPC side changes to have a different codec for server to client communication - Key: HBASE-9136 URL: https://issues.apache.org/jira/browse/HBASE-9136 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 With reference to the mail sent in the dev list, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 We should have a provision such that the codec on the server side could be different from the one on the client side. This would help to remove the tags for security usecases. This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9123) Filter protobuf generated code from long line warning
[ https://issues.apache.org/jira/browse/HBASE-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731065#comment-13731065 ] Hudson commented on HBASE-9123: --- SUCCESS: Integrated in HBase-0.94 #1098 (See [https://builds.apache.org/job/HBase-0.94/1098/]) HBASE-9123 Use table dir modtime to avoid scanning table dir to check cached table descriptor in 0.94 (Dave Latham) (larsh: rev 1511018) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java Filter protobuf generated code from long line warning - Key: HBASE-9123 URL: https://issues.apache.org/jira/browse/HBASE-9123 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.98.0 Attachments: 9123.patch, 9123-v2.patch For big patch, such as the one for namespace, there would be many changes in the protobuf generated code. See example here: https://builds.apache.org/job/PreCommit-HBASE-Build/6569/console We should filter protobuf generated code from long line warning -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9140) NPE in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731069#comment-13731069
]
Elliott Clark commented on HBASE-9140:
--
YCSB runs the clean up on the same thread as the HTable is created. So I don't
think that it's a threading issue when dealing with HTable.
https://github.com/brianfrankcooper/YCSB/blob/master/core/src/main/java/com/yahoo/ycsb/Client.java#L138
NPE in AsyncProcess
---
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9129) Scanner prefetching leaks scanners.
[
https://issues.apache.org/jira/browse/HBASE-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731109#comment-13731109
]
stack commented on HBASE-9129:
--
Commit this and open new issue for lease contention? Should be easy enough to
replicate in a little harness.
Scanner prefetching leaks scanners.
---
Key: HBASE-9129
URL: https://issues.apache.org/jira/browse/HBASE-9129
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.2
Attachments: HBASE-9129-0.patch
Running YCSB workload-e on 0.95 basically hangs the whole cluster.
YCSB opens a scanner for 100 rows.
# YCSB Calls next
# Pre-fetching starts
# YCSB Closes scanner
# Pre-fetching re-adds the scanner.
So the end result is:
{noformat}
scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc
runnable [0x7f75ffefd000]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272)
at java.util.PriorityQueue.indexOf(PriorityQueue.java:342)
at java.util.PriorityQueue.remove(PriorityQueue.java:360)
at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476)
at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232)
- locked 0x7f774455a660 (a java.util.concurrent.DelayQueue)
at
org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269)
at
org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb
waiting on condition [0x7f7605b79000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x7f774455ad78 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba
waiting on condition [0x7f7609ab8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x7f774455ad78 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722){noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731116#comment-13731116 ] Lars Hofhansl edited comment on HBASE-9079 at 8/6/13 7:05 PM: -- So thinking about this again. Why can't we take the largest of any seekHint when we *and* all the filters together in a FilterList? In your case if you add the filters to the list in a different order, will your patch still work? Is the actual problem here that a Filter returns a KV from getNextKeyHint even when it would not have returned SEEK_NEXT_USING_HINT? was (Author: lhofhansl): So thinking about this again. Why can't we take the largest of any seekHint when we *and* all the filters together in a FilterList? In you case if you add the filters to the list in a different order, will your patch still work? Is the actual problem here that a Filter returns a KV from getNextKeyHint even when it would not have return SEEK_NEXT_USING_HINT? FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9079: - Fix Version/s: (was: 0.94.11) 0.94.12 FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731116#comment-13731116 ] Lars Hofhansl commented on HBASE-9079: -- So thinking about this again. Why can't we take the largest of any seekHint when the *and* all the filters together in a FilterList? In you case if you add the filters to the list in a different order, will your patch still work? Is the actual problem here that a Filter returns a KV from getNextKeyHint even when it would not have return SEEK_NEXT_USING_HINT? FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9126) Make HFile MIN VERSION as 2
[ https://issues.apache.org/jira/browse/HBASE-9126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731119#comment-13731119 ] Hudson commented on HBASE-9126: --- SUCCESS: Integrated in hbase-0.95 #408 (See [https://builds.apache.org/job/hbase-0.95/408/]) HBASE-9126 Make HFile MIN VERSION as 2 (anoopsamjohn: rev 1511024) * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java Make HFile MIN VERSION as 2 --- Key: HBASE-9126 URL: https://issues.apache.org/jira/browse/HBASE-9126 Project: HBase Issue Type: Improvement Affects Versions: 0.95.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-9126.patch, HBASE-9126_V2.patch Removed the HFile V1 support from version95. We can make the min version supported as 2? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731116#comment-13731116 ] Lars Hofhansl edited comment on HBASE-9079 at 8/6/13 7:04 PM: -- So thinking about this again. Why can't we take the largest of any seekHint when we *and* all the filters together in a FilterList? In you case if you add the filters to the list in a different order, will your patch still work? Is the actual problem here that a Filter returns a KV from getNextKeyHint even when it would not have return SEEK_NEXT_USING_HINT? was (Author: lhofhansl): So thinking about this again. Why can't we take the largest of any seekHint when the *and* all the filters together in a FilterList? In you case if you add the filters to the list in a different order, will your patch still work? Is the actual problem here that a Filter returns a KV from getNextKeyHint even when it would not have return SEEK_NEXT_USING_HINT? FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Fix For: 0.98.0, 0.95.2, 0.94.11 Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9140) NPE in AsyncProcess
[
https://issues.apache.org/jira/browse/HBASE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731126#comment-13731126
]
Nicolas Liochon commented on HBASE-9140:
you're right. It was too easy. I went through the YCSB code. Nothing fancy.
NPE in AsyncProcess
---
Key: HBASE-9140
URL: https://issues.apache.org/jira/browse/HBASE-9140
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.95.2
Environment: 0.95.2 + Hadoop 2.0.4
Reporter: Elliott Clark
Running YCSB progress got stuck for quite a while and then a Null Pointer
Exception was thrown:
{noformat}
230 sec: 1266253 operations; 0 current ops/sec;
240 sec: 1266253 operations; 0 current ops/sec;
250 sec: 1266253 operations; 0 current ops/sec;
260 sec: 1266253 operations; 0 current ops/sec;
270 sec: 1266253 operations; 0 current ops/sec;
280 sec: 1266253 operations; 0 current ops/sec;
290 sec: 1266253 operations; 0 current ops/sec;
300 sec: 1266253 operations; 0 current ops/sec;
310 sec: 1266253 operations; 0 current ops/sec;
320 sec: 1266253 operations; 0 current ops/sec;
330 sec: 1266253 operations; 0 current ops/sec;
340 sec: 1266253 operations; 0 current ops/sec;
350 sec: 1266253 operations; 0 current ops/sec;
360 sec: 1266253 operations; 0 current ops/sec;
370 sec: 1266253 operations; 0 current ops/sec;
380 sec: 1266253 operations; 0 current ops/sec;
390 sec: 1266253 operations; 0 current ops/sec;
Exception in thread Thread-26 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
400 sec: 1266253 operations; 0 current ops/sec; [UPDATE
AverageLatency(us)=1341720] [INSERT AverageLatency(us)=197941421.07] [CLEANUP
AverageLatency(us)=1342113]
Exception in thread Thread-16 java.lang.NullPointerException
at
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:288)
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:233)
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:811)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1192)
at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:106)
at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
at com.yahoo.ycsb.ClientThread.run(Client.java:307)
410 sec: 1266253 operations; 0 current ops/sec; [INSERT
AverageLatency(us)=208462580.75]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-9141) Replication Znodes Backup Tool
[ https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-9141: --- Attachment: HBase-9141.patch Replication Znodes Backup Tool -- Key: HBASE-9141 URL: https://issues.apache.org/jira/browse/HBASE-9141 Project: HBase Issue Type: Improvement Components: migration, Replication Affects Versions: 0.94.10 Reporter: Himanshu Vashishtha Fix For: 0.95.2 Attachments: HBase-9141.patch While migrating to 0.96, we recommend deleting old znodes so users not face issues like HBASE-7766, and let HBase create them out of box. Though HBase tends to store only ephemeral data in zookeeper, replication has a different approach. Almost all of its data (state, peer info, logs, etc) is present in zookeeper. We would like to preserve them in order to not do re-adding of peers, and ensuring complete replication after we have migrated to 0.96. This jira adds a tool to serialize/de-serialize replication znodes to the underlying filesystem. This could be used while migrating to 0.96.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9079) FilterList getNextKeyHint skips rows that should be included in the results
[ https://issues.apache.org/jira/browse/HBASE-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731130#comment-13731130 ] Viral Bajaria commented on HBASE-9079: -- If we take the largest seekHint, the problem can be explained as follows: - FilterList with FuzzyRow + ColumnRange with MUST_PASS_ALL. - FuzzyRow includes the filter and says move on to ColumnRange. - ColumnRange says first column is not a match and I can give you a better seekHint - FilterList calls seekHint on both FuzzyRow and ColumnRange. FuzzyRow returns the next row that we should use while ColumnRange returns the next column from the originally selected row. If we keep max here then we move on to the next row and do no return the columns from the row that should have been returned. The test case proves that this is what happened originally (though I removed the TestFail.patch due to some Hadoop QA issues) Yes the current changes work even if you change the ordering of filters. The test in the patch verified that behavior too. FilterList getNextKeyHint skips rows that should be included in the results --- Key: HBASE-9079 URL: https://issues.apache.org/jira/browse/HBASE-9079 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.10 Reporter: Viral Bajaria Fix For: 0.98.0, 0.95.2, 0.94.12 Attachments: HBASE-9079-0.94.patch, HBASE-9079-trunk.patch I hit a weird issue/bug and am able to reproduce the error consistently. The problem arises when FilterList has two filters where each implements the getNextKeyHint method. The way the current implementation works is, StoreScanner will call matcher.getNextKeyHint() whenever it gets a SEEK_NEXT_USING_HINT. This in turn will call filter.getNextKeyHint() which at this stage is of type FilterList. The implementation in FilterList iterates through all the filters and keeps the max KeyValue that it sees. All is fine if you wrap filters in FilterList in which only one of them implements getNextKeyHint. but if multiple of them implement then that's where things get weird. For example: - create two filters: one is FuzzyRowFilter and second is ColumnRangeFilter. Both of them implement getNextKeyHint - wrap them in FilterList with MUST_PASS_ALL - FuzzyRowFilter will seek to the correct first row and then pass it to ColumnRangeFilter which will return the SEEK_NEXT_USING_HINT code. - Now in FilterList when getNextKeyHint is called, it calls the one on FuzzyRow first which basically says what the next row should be. While in reality we want the ColumnRangeFilter to give the seek hint. - The above behavior skips data that should be returned, which I have verified by using a RowFilter with RegexStringComparator. I updated the FilterList to maintain state on which filter returns the SEEK_NEXT_USING_HINT and in getNextKeyHint, I invoke the method on the saved filter and reset that state. I tested it with my current queries and it works fine but I need to run the entire test suite to make sure I have not introduced any regression. In addition to that I need to figure out what should be the behavior when the opeation is MUST_PASS_ONE, but I doubt it should be any different. Is my understanding of it being a bug correct ? Or am I trivializing it and ignoring something very important ? If it's tough to wrap your head around the explanation, then I can open a JIRA and upload a patch against 0.94 head. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[
https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731131#comment-13731131
]
Himanshu Vashishtha commented on HBASE-9141:
Patch to add replication znode migration tool. Sample usage:
{code}
bin/hbase org.apache.hadoop.hbase.replication.ReplicationZnodesBackup -h
usage: TakeReplicationZnodesBackup [-d] [-h] [-p arg] [-restore]
-d,--deleteOriginalZnodes Delete the original replication znodes
-h,--help Help
-p,--path arg path where to store the backup file under
hbase.rootdir
-restore,--restoreFromABackup Whether to restore replication znodes
from a backup file
The backup file name is replication-znodes-backup.
In case no parent directory is provided, it stores the replication znodes under
hbase.rootdir.
Example Usage:
Let's say the current root hbase znode is /hbase-94. We want to do a backup
of all the znodes under /hbase-94/replication.
$ $HBASE_HOME/bin/hbase
org.apache.hadoop.hbase.replication.ReplicationZnodesBackup -p myBackup
It will serializes all the znodes under /hbase-94/replication to a
file,replication-znodes-backup under hbase.rootdir, i.e., at
hbase.rootdir/myBackup/replication-znodes-backup
To restore a backup of replication znodes from
myBackup/replication-znodes-backup
$ $HBASE_HOME/bin/hbase
org.apache.hadoop.hbase.replication.ReplicationZnodesBackup -restore -p myBackup
{code}
This could be part of a script to do the migration to 0.96.
Replication Znodes Backup Tool
--
Key: HBASE-9141
URL: https://issues.apache.org/jira/browse/HBASE-9141
Project: HBase
Issue Type: Improvement
Components: migration, Replication
Affects Versions: 0.94.10
Reporter: Himanshu Vashishtha
Fix For: 0.95.2
Attachments: HBase-9141.patch
While migrating to 0.96, we recommend deleting old znodes so users not face
issues like HBASE-7766, and let HBase create them out of box.
Though HBase tends to store only ephemeral data in zookeeper, replication has
a different approach. Almost all of its data (state, peer info, logs, etc) is
present in zookeeper. We would like to preserve them in order to not do
re-adding of peers, and ensuring complete replication after we have migrated
to 0.96.
This jira adds a tool to serialize/de-serialize replication znodes to the
underlying filesystem. This could be used while migrating to 0.96.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8348) Polish the migration to 0.96
[ https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731135#comment-13731135 ] Himanshu Vashishtha commented on HBASE-8348: And, I would argue that HBASE-7766 is one of the reason I would vote for nuking all the znodes and let HBase re-create them. We do need replication znodes to migrate through. I opened HBASE-9141 for that task. Polish the migration to 0.96 Key: HBASE-8348 URL: https://issues.apache.org/jira/browse/HBASE-8348 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Jean-Daniel Cryans Assignee: rajeshbabu Priority: Critical Fix For: 0.95.2 Attachments: HBASE-8348_trunk.patch, HBASE-8348_trunk_v2.patch Currently, migration works but there's still a couple of rough edges: - HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's still on the filesystem. - Data in ZK needs to be removed manually. Either we fix up the data in ZK or we delete it ourselves. - TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but ROOT is gone now. Elliott was also mentioning that we could have hbase migrate do the HFileV1 checks, clear ZK, remove ROOT, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8348) Polish the migration to 0.96
[ https://issues.apache.org/jira/browse/HBASE-8348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731141#comment-13731141 ] Himanshu Vashishtha commented on HBASE-8348: And yes, I agree with disabled table getting enabled in 0.96; I think it is okay if we doc that the table which are disabled in 0.94.x, might get enabled on migration. And, users could disable them once the migration is done. What do you think Rajeshbabu? Polish the migration to 0.96 Key: HBASE-8348 URL: https://issues.apache.org/jira/browse/HBASE-8348 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Jean-Daniel Cryans Assignee: rajeshbabu Priority: Critical Fix For: 0.95.2 Attachments: HBASE-8348_trunk.patch, HBASE-8348_trunk_v2.patch Currently, migration works but there's still a couple of rough edges: - HBASE-8045 finished the .META. migration but didn't remove ROOT, so it's still on the filesystem. - Data in ZK needs to be removed manually. Either we fix up the data in ZK or we delete it ourselves. - TestMetaMigrationRemovingHTD has a testMetaUpdatedFlagInROOT method, but ROOT is gone now. Elliott was also mentioning that we could have hbase migrate do the HFileV1 checks, clear ZK, remove ROOT, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-8627: -- Resolution: Fixed Fix Version/s: 0.95.2 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to Trunk and 0.95 Thanks a lot for the reviews. HBCK can not fix meta not assigned issue Key: HBASE-8627 URL: https://issues.apache.org/jira/browse/HBASE-8627 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.95.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8627_Trunk.patch, HBASE-8627_Trunk-V2.patch When meta table region is not assigned to any RS, HBCK run will get exception. I can see code added in checkMetaRegion() to solve this issue but it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8627) HBCK can not fix meta not assigned issue
[ https://issues.apache.org/jira/browse/HBASE-8627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731142#comment-13731142 ] Anoop Sam John commented on HBASE-8627: --- [~rajesh23] Can you raise a ticket for the issue you have mentioned. HBCK can not fix meta not assigned issue Key: HBASE-8627 URL: https://issues.apache.org/jira/browse/HBASE-8627 Project: HBase Issue Type: Bug Components: hbck Affects Versions: 0.95.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.98.0, 0.95.2 Attachments: HBASE-8627_Trunk.patch, HBASE-8627_Trunk-V2.patch When meta table region is not assigned to any RS, HBCK run will get exception. I can see code added in checkMetaRegion() to solve this issue but it wont work. It still refers to ROOT region! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9129) Scanner prefetching leaks scanners.
[
https://issues.apache.org/jira/browse/HBASE-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731151#comment-13731151
]
Elliott Clark commented on HBASE-9129:
--
Sounds good. You +1 [~stack] ?
Scanner prefetching leaks scanners.
---
Key: HBASE-9129
URL: https://issues.apache.org/jira/browse/HBASE-9129
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.2
Attachments: HBASE-9129-0.patch
Running YCSB workload-e on 0.95 basically hangs the whole cluster.
YCSB opens a scanner for 100 rows.
# YCSB Calls next
# Pre-fetching starts
# YCSB Closes scanner
# Pre-fetching re-adds the scanner.
So the end result is:
{noformat}
scan-prefetch-2-thread-45 daemon prio=10 tid=0x7f7e406ec800 nid=0x40bc
runnable [0x7f75ffefd000]
java.lang.Thread.State: RUNNABLE
at
org.apache.hadoop.hbase.regionserver.Leases$Lease.equals(Leases.java:272)
at java.util.PriorityQueue.indexOf(PriorityQueue.java:342)
at java.util.PriorityQueue.remove(PriorityQueue.java:360)
at java.util.concurrent.DelayQueue.remove(DelayQueue.java:476)
at
org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:232)
- locked 0x7f774455a660 (a java.util.concurrent.DelayQueue)
at
org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:269)
at
org.apache.hadoop.hbase.regionserver.RegionScannerHolder$ScanPrefetcher.call(RegionScannerHolder.java:260)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
scan-prefetch-2-thread-44 daemon prio=10 tid=0x7f7e4c1ba800 nid=0x40bb
waiting on condition [0x7f7605b79000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x7f774455ad78 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
scan-prefetch-2-thread-43 daemon prio=10 tid=0x7f7e38cbc800 nid=0x40ba
waiting on condition [0x7f7609ab8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x7f774455ad78 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722){noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[
https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731153#comment-13731153
]
Himanshu Vashishtha commented on HBASE-9141:
I tested this on a local filesystem and hdfs with some replication znodes.
A successful Backup run:
{code}
bin/hbase org.apache.hadoop.hbase.replication.ReplicationZnodesBackup -p
myBackup
Backup Znodes file is:
hdfs://localhost:41020/hbase-0.94/myBackup/replication-znodes-backup
SUCCESSFULLY SERIALIZED 3 replication znodes.
On the zkcli console, I do a ls of replication znode:
{code}
[zk: localhost:2181(CONNECTED) 1] ls /hbase/replication
[peers, rs]
{code}
Then, I delete the /hbase/replication znode
{code}
[zk: localhost:2181(CONNECTED) 3] rmr /hbase/replication
[zk: localhost:2181(CONNECTED) 4] ls /hbase/replication
Node does not exist: /hbase/replication
{code}
Then, I restore the znode using the backup script. A successful Restore run:
{code}
bin/hbase org.apache.hadoop.hbase.replication.ReplicationZnodesBackup -restore
-p myBackup
Backup Znodes file is:
hdfs://localhost:41020/hbase-0.94/myBackup/replication-znodes-backup
SUCCESSFULLY RESTORED 3 replication znodes.
{code}
{code}
[zk: localhost:2181(CONNECTED) 5] ls /hbase/replication
[peers, rs]
{code}
Replication Znodes Backup Tool
--
Key: HBASE-9141
URL: https://issues.apache.org/jira/browse/HBASE-9141
Project: HBase
Issue Type: Improvement
Components: migration, Replication
Affects Versions: 0.94.10
Reporter: Himanshu Vashishtha
Fix For: 0.95.2
Attachments: HBase-9141.patch
While migrating to 0.96, we recommend deleting old znodes so users not face
issues like HBASE-7766, and let HBase create them out of box.
Though HBase tends to store only ephemeral data in zookeeper, replication has
a different approach. Almost all of its data (state, peer info, logs, etc) is
present in zookeeper. We would like to preserve them in order to not do
re-adding of peers, and ensuring complete replication after we have migrated
to 0.96.
This jira adds a tool to serialize/de-serialize replication znodes to the
underlying filesystem. This could be used while migrating to 0.96.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9136) RPC side changes to have a different codec for server to client communication
[ https://issues.apache.org/jira/browse/HBASE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731154#comment-13731154 ] Anoop Sam John commented on HBASE-9136: --- In both cases (Return scan results to client or replication) KVs are serialized by RpcServer by getting the payload(cells) from the PayloadCarryingRpcController. So not sure how easy it will be, to do any such differentiation in the RPC layer. JFYI RPC side changes to have a different codec for server to client communication - Key: HBASE-9136 URL: https://issues.apache.org/jira/browse/HBASE-9136 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 With reference to the mail sent in the dev list, http://comments.gmane.org/gmane.comp.java.hadoop.hbase.devel/38984 We should have a provision such that the codec on the server side could be different from the one on the client side. This would help to remove the tags for security usecases. This JIRA is aimed to provide that capability in the codec itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-9141) Replication Znodes Backup Tool
[
https://issues.apache.org/jira/browse/HBASE-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731160#comment-13731160
]
Himanshu Vashishtha commented on HBASE-9141:
Doh! Can some one with edit-comment control could please edit the formatting of
above comment? Somehow, I missed one \{code\} symbol above. :)
Thanks.
Replication Znodes Backup Tool
--
Key: HBASE-9141
URL: https://issues.apache.org/jira/browse/HBASE-9141
Project: HBase
Issue Type: Improvement
Components: migration, Replication
Affects Versions: 0.94.10
Reporter: Himanshu Vashishtha
Fix For: 0.95.2
Attachments: HBase-9141.patch
While migrating to 0.96, we recommend deleting old znodes so users not face
issues like HBASE-7766, and let HBase create them out of box.
Though HBase tends to store only ephemeral data in zookeeper, replication has
a different approach. Almost all of its data (state, peer info, logs, etc) is
present in zookeeper. We would like to preserve them in order to not do
re-adding of peers, and ensuring complete replication after we have migrated
to 0.96.
This jira adds a tool to serialize/de-serialize replication znodes to the
underlying filesystem. This could be used while migrating to 0.96.0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
