[jira] [Commented] (HBASE-7789) Clean DeadServer.java and add a Jitter method in ConnectionUtils
[ https://issues.apache.org/jira/browse/HBASE-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575692#comment-13575692 ] nkeywal commented on HBASE-7789: Ok for the comment, will do that in the commit (tomorrow) if there are not other reviews. > Clean DeadServer.java and add a Jitter method in ConnectionUtils > > > Key: HBASE-7789 > URL: https://issues.apache.org/jira/browse/HBASE-7789 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Fix For: 0.96.0 > > Attachments: 7789.v1.patch > > > I need to do some changes in DeadServer because of HBASE-7390. To minimize > the patch size and simplifies the feedback, I prefer to isolate the issue. > Changes are: > - Add the time when the server was declared as dead. It's what I need in > HBASE-7390, but it makes sense even without it, for example to be shown in > the UI. > - suppress the extends on Set & clean up all the not used methods > - use directly the object instead of a copy. > For connection utils, we currently have a jitter of 1%. I need a bigger one > for sure in one case, but I wonder if we should not increase it in all cases? > instead of plus 1%, we should have plus or minus 10% imho. > Tests are in progress locally, I will add the patch when they're ok. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7462) TestDrainingServer is an integration test. It should be a unit test instead
[ https://issues.apache.org/jira/browse/HBASE-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575694#comment-13575694 ] nkeywal edited comment on HBASE-7462 at 2/11/13 9:22 AM: - We should not mock up the ZK itself, just the calls to ZK for this scenario. Ideally again, as it easier said than done :-) was (Author: nkeywal): We should not mock up the ZK itself, just the calls to ZK for this scenario. Ideally again, as it easier to say that to do :-) > TestDrainingServer is an integration test. It should be a unit test instead > --- > > Key: HBASE-7462 > URL: https://issues.apache.org/jira/browse/HBASE-7462 > Project: HBase > Issue Type: Wish > Components: test >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Trivial > Labels: noob > > TestDrainingServer tests the function that allows to say that a regionserver > should not get new regions. > As it is written today, it's an integration test: it starts & stops a cluster. > The test would be more efficient if it would just check that the > AssignmentManager does not use the drained region server; whatever the > circumstances (bulk assign or not for example). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7462) TestDrainingServer is an integration test. It should be a unit test instead
[ https://issues.apache.org/jira/browse/HBASE-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575694#comment-13575694 ] nkeywal commented on HBASE-7462: We should not mock up the ZK itself, just the calls to ZK for this scenario. Ideally again, as it easier to say that to do :-) > TestDrainingServer is an integration test. It should be a unit test instead > --- > > Key: HBASE-7462 > URL: https://issues.apache.org/jira/browse/HBASE-7462 > Project: HBase > Issue Type: Wish > Components: test >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Trivial > Labels: noob > > TestDrainingServer tests the function that allows to say that a regionserver > should not get new regions. > As it is written today, it's an integration test: it starts & stops a cluster. > The test would be more efficient if it would just check that the > AssignmentManager does not use the drained region server; whatever the > circumstances (bulk assign or not for example). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575697#comment-13575697 ] Anoop Sam John commented on HBASE-7801: --- bq.For batch operation we'd take the safest sync option of any of the mutations. I.e. if there is at least one that wants to be flushed we'd sync the batch, if there's none of those but at least one that wants deferred flush we defer flush the batch This also to be default to defered = false When the attribute is missing we can assume the value to be that set for the table? When the attribute is there we go by that for the Mutation. And when atleast one need WAL to be synced immediately , go with that. If all need WAL to be defered synced or none have this attribute but at the table level it is defered, then go with defered. [~lhofhansl] Here batch will be the mini batch. Which all Mutations will come under one mini batch depends on the row locking. Another point not directly related to this issue. Now we have the LogSyncer thread started and running. This thread can come at any point during the put and can do a sync. If all my writes I want to be defered sync = false, this thread may be of no real use for me. In our case of secondary indexing we wanted the sync to happen as one operation (sync write for actual region and that for index region). This is possible but the issue is in between the LogSyncer thread can come and do a sync. So should we make a way to so as to control the start of LogSyncer ? > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7789) Clean DeadServer.java and add a Jitter method in ConnectionUtils
[ https://issues.apache.org/jira/browse/HBASE-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-7789: --- Description: I need to do some changes in DeadServer because of HBASE-7590. To minimize the patch size and simplifies the feedback, I prefer to isolate the issue. Changes are: - Add the time when the server was declared as dead. It's what I need in HBASE-7590, but it makes sense even without it, for example to be shown in the UI. - suppress the extends on Set & clean up all the not used methods - use directly the object instead of a copy. For connection utils, we currently have a jitter of 1%. I need a bigger one for sure in one case, but I wonder if we should not increase it in all cases? instead of plus 1%, we should have plus or minus 10% imho. Tests are in progress locally, I will add the patch when they're ok. was: I need to do some changes in DeadServer because of HBASE-7390. To minimize the patch size and simplifies the feedback, I prefer to isolate the issue. Changes are: - Add the time when the server was declared as dead. It's what I need in HBASE-7390, but it makes sense even without it, for example to be shown in the UI. - suppress the extends on Set & clean up all the not used methods - use directly the object instead of a copy. For connection utils, we currently have a jitter of 1%. I need a bigger one for sure in one case, but I wonder if we should not increase it in all cases? instead of plus 1%, we should have plus or minus 10% imho. Tests are in progress locally, I will add the patch when they're ok. > Clean DeadServer.java and add a Jitter method in ConnectionUtils > > > Key: HBASE-7789 > URL: https://issues.apache.org/jira/browse/HBASE-7789 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal > Fix For: 0.96.0 > > Attachments: 7789.v1.patch > > > I need to do some changes in DeadServer because of HBASE-7590. To minimize > the patch size and simplifies the feedback, I prefer to isolate the issue. > Changes are: > - Add the time when the server was declared as dead. It's what I need in > HBASE-7590, but it makes sense even without it, for example to be shown in > the UI. > - suppress the extends on Set & clean up all the not used methods > - use directly the object instead of a copy. > For connection utils, we currently have a jitter of 1%. I need a bigger one > for sure in one case, but I wonder if we should not increase it in all cases? > instead of plus 1%, we should have plus or minus 10% imho. > Tests are in progress locally, I will add the patch when they're ok. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7407) TestMasterFailover under tests some cases and over tests some others
[
https://issues.apache.org/jira/browse/HBASE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-7407:
---
Status: Open (was: Patch Available)
> TestMasterFailover under tests some cases and over tests some others
>
>
> Key: HBASE-7407
> URL: https://issues.apache.org/jira/browse/HBASE-7407
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment, test
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 7407.v1.patch, 7407.v2.patch, 7407.v3.patch,
> 7407.v5.patch, 7407.v6.patch, 7407_v7.patch
>
>
> The tests are done with this settings:
> conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000);
> conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 4000);
> As a results:
> 1) some tests seems to work, but in real life, the recovery would take 5
> minutes or more, as in production there always higher. So we don't see the
> real issues.
> 2) The tests include specific cases that should not happen in production. It
> works because the timeout catches everything, but these scenarios do not need
> to be optimized, as they cannot happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7407) TestMasterFailover under tests some cases and over tests some others
[
https://issues.apache.org/jira/browse/HBASE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-7407:
---
Attachment: 7407_v7.patch
> TestMasterFailover under tests some cases and over tests some others
>
>
> Key: HBASE-7407
> URL: https://issues.apache.org/jira/browse/HBASE-7407
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment, test
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 7407.v1.patch, 7407.v2.patch, 7407.v3.patch,
> 7407.v5.patch, 7407.v6.patch, 7407_v7.patch
>
>
> The tests are done with this settings:
> conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000);
> conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 4000);
> As a results:
> 1) some tests seems to work, but in real life, the recovery would take 5
> minutes or more, as in production there always higher. So we don't see the
> real issues.
> 2) The tests include specific cases that should not happen in production. It
> works because the timeout catches everything, but these scenarios do not need
> to be optimized, as they cannot happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7407) TestMasterFailover under tests some cases and over tests some others
[
https://issues.apache.org/jira/browse/HBASE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-7407:
---
Status: Patch Available (was: Open)
> TestMasterFailover under tests some cases and over tests some others
>
>
> Key: HBASE-7407
> URL: https://issues.apache.org/jira/browse/HBASE-7407
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment, test
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 7407.v1.patch, 7407.v2.patch, 7407.v3.patch,
> 7407.v5.patch, 7407.v6.patch, 7407_v7.patch
>
>
> The tests are done with this settings:
> conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000);
> conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 4000);
> As a results:
> 1) some tests seems to work, but in real life, the recovery would take 5
> minutes or more, as in production there always higher. So we don't see the
> real issues.
> 2) The tests include specific cases that should not happen in production. It
> works because the timeout catches everything, but these scenarios do not need
> to be optimized, as they cannot happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7812) script
Samir Ahmic created HBASE-7812: -- Summary: script Key: HBASE-7812 URL: https://issues.apache.org/jira/browse/HBASE-7812 Project: HBase Issue Type: Bug Reporter: Samir Ahmic -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7812) script
[ https://issues.apache.org/jira/browse/HBASE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7812: --- Priority: Trivial (was: Major) > script > -- > > Key: HBASE-7812 > URL: https://issues.apache.org/jira/browse/HBASE-7812 > Project: HBase > Issue Type: Bug >Reporter: Samir Ahmic >Priority: Trivial > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7812) script to return servers where META and ROOT tables are deployed
[ https://issues.apache.org/jira/browse/HBASE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7812: --- Issue Type: Wish (was: Bug) Summary: script to return servers where META and ROOT tables are deployed (was: script) > script to return servers where META and ROOT tables are deployed > > > Key: HBASE-7812 > URL: https://issues.apache.org/jira/browse/HBASE-7812 > Project: HBase > Issue Type: Wish >Reporter: Samir Ahmic >Priority: Trivial > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7812) script to return servers where META and ROOT tables are deployed
[ https://issues.apache.org/jira/browse/HBASE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575784#comment-13575784 ] Jean-Marc Spaggiari commented on HBASE-7812: Hi Samir, What do you mean by "a script"? Are you talking about a bash script? Or something on the HBase shell? Or do you need the java piece of code to do that? > script to return servers where META and ROOT tables are deployed > > > Key: HBASE-7812 > URL: https://issues.apache.org/jira/browse/HBASE-7812 > Project: HBase > Issue Type: Wish >Reporter: Samir Ahmic >Priority: Trivial > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7812) script to return servers where META and ROOT tables are deployed
[ https://issues.apache.org/jira/browse/HBASE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575790#comment-13575790 ] Samir Ahmic commented on HBASE-7812: Hi, JM It would be nice to have bash or ruby script that is doing this job. > script to return servers where META and ROOT tables are deployed > > > Key: HBASE-7812 > URL: https://issues.apache.org/jira/browse/HBASE-7812 > Project: HBase > Issue Type: Wish >Reporter: Samir Ahmic >Priority: Trivial > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7812) script to return servers where META and ROOT tables are deployed
[ https://issues.apache.org/jira/browse/HBASE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7812: --- Description: Simple script (bash or ruby) that would print servers addresses(hostnames) where ROOT or META tables are deployed. It could be located in bin dir. I have attached simple example that i use for this. > script to return servers where META and ROOT tables are deployed > > > Key: HBASE-7812 > URL: https://issues.apache.org/jira/browse/HBASE-7812 > Project: HBase > Issue Type: Wish >Reporter: Samir Ahmic >Priority: Trivial > > Simple script (bash or ruby) that would print servers addresses(hostnames) > where ROOT or META tables are deployed. It could be located in bin dir. I > have attached simple example that i use for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7812) script to return servers where META and ROOT tables are deployed
[ https://issues.apache.org/jira/browse/HBASE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7812: --- Attachment: HBASE-7812.txt > script to return servers where META and ROOT tables are deployed > > > Key: HBASE-7812 > URL: https://issues.apache.org/jira/browse/HBASE-7812 > Project: HBase > Issue Type: Wish >Reporter: Samir Ahmic >Priority: Trivial > Attachments: HBASE-7812.txt > > > Simple script (bash or ruby) that would print servers addresses(hostnames) > where ROOT or META tables are deployed. It could be located in bin dir. I > have attached simple example that i use for this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7407) TestMasterFailover under tests some cases and over tests some others
[
https://issues.apache.org/jira/browse/HBASE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575801#comment-13575801
]
Hadoop QA commented on HBASE-7407:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568817/7407_v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4410//console
This message is automatically generated.
> TestMasterFailover under tests some cases and over tests some others
>
>
> Key: HBASE-7407
> URL: https://issues.apache.org/jira/browse/HBASE-7407
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment, test
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Minor
> Attachments: 7407.v1.patch, 7407.v2.patch, 7407.v3.patch,
> 7407.v5.patch, 7407.v6.patch, 7407_v7.patch
>
>
> The tests are done with this settings:
> conf.setInt("hbase.master.assignment.timeoutmonitor.period", 2000);
> conf.setInt("hbase.master.assignment.timeoutmonitor.timeout", 4000);
> As a results:
> 1) some tests seems to work, but in real life, the recovery would take 5
> minutes or more, as in production there always higher. So we don't see the
> real issues.
> 2) The tests include specific cases that should not happen in production. It
> works because the timeout catches everything, but these scenarios do not need
> to be optimized, as they cannot happen.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-6132:
--
Attachment: HBASE-6132_94.patch
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Attachments: HBASE-6132_94.patch, TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-6132:
--
Attachment: HBASE-6132_Trunk.patch
Let QA run the trunk patch
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Attachments: HBASE-6132_94.patch, HBASE-6132_Trunk.patch,
> TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-6132:
--
Fix Version/s: 0.94.6
0.96.0
Status: Patch Available (was: Open)
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.94.0, 0.92.1, 0.92.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-6132_94.patch, HBASE-6132_Trunk.patch,
> TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575883#comment-13575883 ] Nicolas Spiegelberg commented on HBASE-7667: Some thoughts I had about this: Overall, I think it's a good idea. Seems like it's not crazy to add and would have multiple benefits. Logical striping across the L1 boundary is a simple solution to both proactively handle splits and reduce compaction times. Thoughts on this feature 1. Fixed configs : in the same way that we got a lot of stability by limiting the regions/server to a fixed number, we might want to similarly limit the number of stripes per region to 10 (or X) instead of "every Y bytes". This will help us understand the benefit we get from striping and it's easy to double the striping and chart the difference. 2. NameNode pressure : Obviously, a 10x striping factor will cause 10x scaling of the FS. Can we offset this by increasing the HDFS block size, since addBlock dominates at scale? Really, unlike Hadoop, you have all of the HFile or none of it. Missing a portion of the HFile currently invalidates the whole file. You really need 1 HDFS block == 1 HFile. However, we could probably just toy with increasing it by the striping factor right now and seeing if that balances things. 3. Open Times : I think this will be an issue, specifically on server start. Need to be careful here. 4. Major compaction : you can perform a major compaction (remove deletes) as long as you have [i,end) contiguous. I don't think you'd need to involve L0 files in an MC at all. Save the complexity. Furthermore, part of the reason why we created the tiered compaction is to prevent small/new files from participating in MC because of cache thrashing, poor minor compactions, and a handful of other reasons. So, some thoughts on related pain points we seem to have that tie into this feature: 1. Reduce Cache thrashing : region moves kill us a lot of time because we have a cold cache. There is a worry that more aggressive compactions mean more thrashing. I think it will actual even this out better since right now a MC causes a lot of churn. Just should be thinking about this if perf after the feature isn't what we desire. 2. Unnecessary IOPS : outside of this algorithm, we should just completely get rid of the requirement to compact after a split. We have the block cache, so given a [start,end) in the file, we can easily tell our mid point for future splits. There's little reason to aggressively churn in this way after splitting. 3. Poor locality : for grid topology setups, we should eventually make the striping algorithm a little more intelligent about picking our replicas. If all stripes go to the same secondary & tertiary node, then splits have a very restricted set of servers to chose for datanode locality. > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no
[jira] [Created] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
Varun Sharma created HBASE-7813: --- Summary: Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes Key: HBASE-7813 URL: https://issues.apache.org/jira/browse/HBASE-7813 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.94.4, 0.96.0 Reporter: Varun Sharma Assignee: Varun Sharma Fix For: 0.96.0, 0.94.5 Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and give it a set of Columns which are not in the row. A Delete is constructed using just the row alone. No family/column/version is added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575894#comment-13575894 ] Anoop Sam John commented on HBASE-7813: --- How is your Scan object looks like? You mean the Scan object contains some columns which are not at all present in row. Still the region scan returns some KVs for that row? > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.5 > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575897#comment-13575897 ] Varun Sharma commented on HBASE-7813: - Code follow for BulkDeleteEndpoint: 1) Delete delete = new Delete(row); 2) Find stuff which is present and then do deleteColumn and deleteColumns and deleteFamily accordingly 3) Apply delete If only 1) and 3) execute, entire row is deleted. If this makes sense, I have a patch which should resolve this, I am seeing this in my setup. Thanks ! > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.5 > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575908#comment-13575908 ] Ted Yu commented on HBASE-7813: --- I think before executing the deletion in step 3, we should check for row deletion. @Varun: Can you upload your patch ? > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.5 > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Sharma updated HBASE-7813: Status: Patch Available (was: Open) > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.94.4, 0.96.0 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.5 > > Attachments: 7813-0.94.txt > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Sharma updated HBASE-7813: Attachment: 7813-0.94.txt Attaching for 0.94, can do trunk if this looks okay. > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.5 > > Attachments: 7813-0.94.txt > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[
https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575912#comment-13575912
]
Hadoop QA commented on HBASE-7813:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568849/7813-0.94.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4412//console
This message is automatically generated.
> Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
> -
>
> Key: HBASE-7813
> URL: https://issues.apache.org/jira/browse/HBASE-7813
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 0.96.0, 0.94.4
>Reporter: Varun Sharma
>Assignee: Varun Sharma
> Fix For: 0.96.0, 0.94.5
>
> Attachments: 7813-0.94.txt
>
>
> Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and
> give it a set of Columns which are not in the row.
> A Delete is constructed using just the row alone. No family/column/version is
> added and it is applied, thus deleting the entire row.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7679) implement store file management for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575917#comment-13575917 ] Sergey Shelukhin commented on HBASE-7679: - posted > implement store file management for stripe compactions > -- > > Key: HBASE-7679 > URL: https://issues.apache.org/jira/browse/HBASE-7679 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7667-and-7603-v0-incomplete.patch, > HBASE-7667-and-7603-v0-incomplete.patch, HBASE-7667-and-7603-v1.patch, > HBASE-7667-and-7603-v1.patch, HBASE-7667-v1.patch, HBASE-7667-v1.patch, > HBASE-7667-v2.patch, HBASE-7667-v2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[
https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575919#comment-13575919
]
ramkrishna.s.vasudevan commented on HBASE-7521:
---
{code}
if ((state.getServerName() != null) && state.getServerName().equals(sn)) {
{code}
This new change has been added because when in OFFLINE state we don't populate
the servername right?
{code}
+ ", which we are already trying to " + " ,which we are already trying to "
{code}
This msg is coming twice.
{code}
try {
addRegionsInTransition(region, CLOSE);
} catch (RegionAlreadyInTransitionException rate) {
LOG.warn("Received close for region we are already opening or closing; "
+ region.getEncodedName());
return false;
}
{code}
Can we throw the exception instead of catching it as in openRegion()?
Rest looks the same.
@Jimmy
Prior to your changes in Trunk wrt to AM things were messy. But i feel that if
we get it in atleast the problem addressed by this JIRA is solved. There may
be some issues hiding some where but the issue addressed here is pretty much
big.
Sergey's integration suite is a good one that tries to catch up many corner
cases. I am +1 on getting this to 0.94.
> fix HBASE-6060 (regions stuck in opening state) in 0.94
> ---
>
> Key: HBASE-7521
> URL: https://issues.apache.org/jira/browse/HBASE-7521
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.94.6
>
> Attachments: HBASE-7521-original-patch-ported-v0.patch,
> HBASE-7521-original-patch-ported-v1.patch,
> HBASE-7521-original-patch-ported-v2.patch,
> HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch,
> HBASE-7521-v1.patch
>
>
> Discussion in HBASE-6060 implies that the fix there does not work on 0.94.
> Still, we may want to fix the issue in 0.94 (via some different fix) because
> the regions stuck in opening for ridiculous amounts of time is not a good
> thing to have.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7799) reassigning region stuck in open still may not work correctly due to leftover ZK node
[
https://issues.apache.org/jira/browse/HBASE-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575921#comment-13575921
]
Sergey Shelukhin commented on HBASE-7799:
-
I'd think so, but it looks like it didn't work, master says that node already
exists and doesn't indicate that it's overwritten.
> reassigning region stuck in open still may not work correctly due to leftover
> ZK node
> -
>
> Key: HBASE-7799
> URL: https://issues.apache.org/jira/browse/HBASE-7799
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> (logs grepped by region name, and abridged.
> META server was dead so OpenRegionHandler for the region took a while, and
> was interrupted:
> {code}
> 2013-02-08 14:35:01,555 DEBUG
> [RS_OPEN_REGION-10.11.2.92,64485,1360362800564-2]
> handler.OpenRegionHandler(255): Interrupting thread
> Thread[PostOpenDeployTasks:871d1c3bdf98a2c93b527cb6cc61327d,5,main]
> {code}
> Then master tried to force region offline and reassign:
> {code}
> 2013-02-08 14:35:06,500 INFO
> [MASTER_SERVER_OPERATIONS-10.11.2.92,64483,1360362800340-1]
> master.RegionStates(347): Found opening region
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OPENING, ts=1360362901596, server=10.11.2.92,64485,1360362800564} to
> be reassigned by SSH for 10.11.2.92,64485,1360362800564
> 2013-02-08 14:35:06,500 INFO
> [MASTER_SERVER_OPERATIONS-10.11.2.92,64483,1360362800340-1]
> master.RegionStates(242): Region {NAME =>
> 'IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.',
> STARTKEY => '732c', ENDKEY => '7ff8', ENCODED =>
> 871d1c3bdf98a2c93b527cb6cc61327d,} transitioned from
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OPENING, ts=1360362901596, server=10.11.2.92,64485,1360362800564} to
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=CLOSED, ts=1360362906500, server=null}
> 2013-02-08 14:35:06,505 DEBUG
> [10.11.2.92,64483,1360362800340-GeneralBulkAssigner-1]
> master.AssignmentManager(1530): Forcing OFFLINE;
> was={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=CLOSED, ts=1360362906500, server=null}
> 2013-02-08 14:35:06,506 DEBUG
> [10.11.2.92,64483,1360362800340-GeneralBulkAssigner-1]
> zookeeper.ZKAssign(176): master:64483-0x13cbbf1025d Async create of
> unassigned node for 871d1c3bdf98a2c93b527cb6cc61327d with OFFLINE state
> {code}
> But didn't delete the original ZK node?
> {code}
> 2013-02-08 14:35:06,509 WARN [main-EventThread] master.OfflineCallback(59):
> Node for /hbase/region-in-transition/871d1c3bdf98a2c93b527cb6cc61327d already
> exists
> 2013-02-08 14:35:06,509 DEBUG [main-EventThread] master.OfflineCallback(69):
> rs={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OFFLINE, ts=1360362906506, server=null},
> server=10.11.2.92,64488,1360362800651
> 2013-02-08 14:35:06,512 DEBUG [main-EventThread]
> master.OfflineCallback$ExistCallback(106):
> rs={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OFFLINE, ts=1360362906506, server=null},
> server=10.11.2.92,64488,1360362800651
> {code}
> So it went into infinite cycle of failing to assign due to this:
> {code}
> 2013-02-08 14:35:06,517 INFO [PRI IPC Server handler 7 on 64488]
> regionserver.HRegionServer(3435): Received request to open region:
> IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> on 10.11.2.92,64488,1360362800651
> 2013-02-08 14:35:06,521 WARN
> [RS_OPEN_REGION-10.11.2.92,64488,1360362800651-0] zookeeper.ZKAssign(762):
> regionserver:64488-0x13cbbf1025d0004 Attempt to transition the unassigned
> node for 871d1c3bdf98a2c93b527cb6cc61327d from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING failed, the node existed but was in the state
> RS_ZK_REGION_OPENING set by the server [wrong server name redacted, see
> HBASE-7798]
> {code}
> Transitioning failed-to-open similarly fails.
> It seems like master needs to nuke ZK node unconditionally to offline?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575924#comment-13575924
]
ramkrishna.s.vasudevan commented on HBASE-6132:
---
Patch looks good. Lets wait for QA.
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-6132_94.patch, HBASE-6132_Trunk.patch,
> TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
Ted Yu created HBASE-7814: - Summary: Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 Key: HBASE-7814 URL: https://issues.apache.org/jira/browse/HBASE-7814 Project: HBase Issue Type: Bug Reporter: Ted Yu Fix For: 0.94.6 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7799) reassigning region stuck in open still may not work correctly due to leftover ZK node
[
https://issues.apache.org/jira/browse/HBASE-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575925#comment-13575925
]
ramkrishna.s.vasudevan commented on HBASE-7799:
---
Okie. If you can attach the logs it would be great. I can take a more closer
look at this if you don't mind.
> reassigning region stuck in open still may not work correctly due to leftover
> ZK node
> -
>
> Key: HBASE-7799
> URL: https://issues.apache.org/jira/browse/HBASE-7799
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> (logs grepped by region name, and abridged.
> META server was dead so OpenRegionHandler for the region took a while, and
> was interrupted:
> {code}
> 2013-02-08 14:35:01,555 DEBUG
> [RS_OPEN_REGION-10.11.2.92,64485,1360362800564-2]
> handler.OpenRegionHandler(255): Interrupting thread
> Thread[PostOpenDeployTasks:871d1c3bdf98a2c93b527cb6cc61327d,5,main]
> {code}
> Then master tried to force region offline and reassign:
> {code}
> 2013-02-08 14:35:06,500 INFO
> [MASTER_SERVER_OPERATIONS-10.11.2.92,64483,1360362800340-1]
> master.RegionStates(347): Found opening region
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OPENING, ts=1360362901596, server=10.11.2.92,64485,1360362800564} to
> be reassigned by SSH for 10.11.2.92,64485,1360362800564
> 2013-02-08 14:35:06,500 INFO
> [MASTER_SERVER_OPERATIONS-10.11.2.92,64483,1360362800340-1]
> master.RegionStates(242): Region {NAME =>
> 'IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.',
> STARTKEY => '732c', ENDKEY => '7ff8', ENCODED =>
> 871d1c3bdf98a2c93b527cb6cc61327d,} transitioned from
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OPENING, ts=1360362901596, server=10.11.2.92,64485,1360362800564} to
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=CLOSED, ts=1360362906500, server=null}
> 2013-02-08 14:35:06,505 DEBUG
> [10.11.2.92,64483,1360362800340-GeneralBulkAssigner-1]
> master.AssignmentManager(1530): Forcing OFFLINE;
> was={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=CLOSED, ts=1360362906500, server=null}
> 2013-02-08 14:35:06,506 DEBUG
> [10.11.2.92,64483,1360362800340-GeneralBulkAssigner-1]
> zookeeper.ZKAssign(176): master:64483-0x13cbbf1025d Async create of
> unassigned node for 871d1c3bdf98a2c93b527cb6cc61327d with OFFLINE state
> {code}
> But didn't delete the original ZK node?
> {code}
> 2013-02-08 14:35:06,509 WARN [main-EventThread] master.OfflineCallback(59):
> Node for /hbase/region-in-transition/871d1c3bdf98a2c93b527cb6cc61327d already
> exists
> 2013-02-08 14:35:06,509 DEBUG [main-EventThread] master.OfflineCallback(69):
> rs={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OFFLINE, ts=1360362906506, server=null},
> server=10.11.2.92,64488,1360362800651
> 2013-02-08 14:35:06,512 DEBUG [main-EventThread]
> master.OfflineCallback$ExistCallback(106):
> rs={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OFFLINE, ts=1360362906506, server=null},
> server=10.11.2.92,64488,1360362800651
> {code}
> So it went into infinite cycle of failing to assign due to this:
> {code}
> 2013-02-08 14:35:06,517 INFO [PRI IPC Server handler 7 on 64488]
> regionserver.HRegionServer(3435): Received request to open region:
> IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> on 10.11.2.92,64488,1360362800651
> 2013-02-08 14:35:06,521 WARN
> [RS_OPEN_REGION-10.11.2.92,64488,1360362800651-0] zookeeper.ZKAssign(762):
> regionserver:64488-0x13cbbf1025d0004 Attempt to transition the unassigned
> node for 871d1c3bdf98a2c93b527cb6cc61327d from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING failed, the node existed but was in the state
> RS_ZK_REGION_OPENING set by the server [wrong server name redacted, see
> HBASE-7798]
> {code}
> Transitioning failed-to-open similarly fails.
> It seems like master needs to nuke ZK node unconditionally to offline?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575589#comment-13575589 ] Jimmy Xiang edited comment on HBASE-7667 at 2/11/13 5:36 PM: - That's right. To major compact a stripe, all L0 files, if any, can be compacted and split into stripes, then merge/compact all files belonging to the stripe. was (Author: jxiang): That's right. To major compact a stripe, all L0 files, if any, can be split into stripes, then merge all files belonging to the stripe. > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575927#comment-13575927
]
Hadoop QA commented on HBASE-6132:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568841/HBASE-6132_Trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestJoinedScanners
org.apache.hadoop.hbase.client.TestHTableMultiplexer
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4411//console
This message is automatically generated.
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-6132_94.patch, HBASE-6132_Trunk.patch,
> TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[
https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575929#comment-13575929
]
Ted Yu commented on HBASE-7813:
---
When I ran unit test, I got:
{code}
testBulkDeleteEndpoint(org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol)
Time elapsed: 40.614 sec <<< ERROR!
java.lang.reflect.UndeclaredThrowableException
at $Proxy22.delete(Unknown Source)
at
org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol$1.call(TestBulkDeleteProtocol.java:124)
at
org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol$1.call(TestBulkDeleteProtocol.java:122)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1463)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
after attempts=10, exceptions:
Mon Feb 11 09:33:35 PST 2013,
org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@40f186b8, java.io.IOException:
java.io.IOException: java.lang.ClassCastException: [Ljava.lang.Object; cannot
be cast to [Lorg.apache.hadoop.hbase.util.Pair;
...
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
... 9 more
{code}
Do you see similar failure ?
> Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
> -
>
> Key: HBASE-7813
> URL: https://issues.apache.org/jira/browse/HBASE-7813
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 0.96.0, 0.94.4
>Reporter: Varun Sharma
>Assignee: Varun Sharma
> Fix For: 0.96.0, 0.94.5
>
> Attachments: 7813-0.94.txt
>
>
> Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and
> give it a set of Columns which are not in the row.
> A Delete is constructed using just the row alone. No family/column/version is
> added and it is applied, thus deleting the entire row.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575934#comment-13575934 ] Jimmy Xiang commented on HBASE-7667: bq. I don't think you'd need to involve L0 files in an MC at all. Agree. The major compaction is not the traditional one any more. We can limit the number of stripes per region. But we don't have to start with that many stripes at the beginning. One thing I'd like to mention is that auto-split a stripe will be much simpler than auto-split a region; auto-merge couple stripes will be much simpler than auto-merge couple regions too. I'd see we should prefer more stripes than more regions, for a given table. > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7815) Too subtile behavior for HConnection#getRegionLocation reload parameter and performance risk
nkeywal created HBASE-7815: -- Summary: Too subtile behavior for HConnection#getRegionLocation reload parameter and performance risk Key: HBASE-7815 URL: https://issues.apache.org/jira/browse/HBASE-7815 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Priority: Minor HConnection#getRegionLocation(table, row, reload=true) and HConnection#getRegionLocation(table, row, reload=false) are not equivalent when the cache is empty: the first will check the table status while the second will not. As a consequence, the client won't have the same exception if the table is disabled. With reload==true, we will have a DoNotRetryIOException, with a message saying that the table is disabled. With reload==false we will have a NotServingException. It's quite difficult to guess, as it's not mentioned in the javadoc. Second effect is that the client is going to ZooKeeper to check this table state. In ServerCallable, if the first try is not successful, we will then go all the time to ZK to check this status. So if a region server stops, all its clients will connect to ZK, possibly multiple time if the recovery takes some time. With a few hundreds clients, it's not very nice... I'm not sure of the solution. A possible improvement in ServerCallable would be to do a reload only at the first retry instead of all of them, but: - it's not without side effects, even if it's limited - the real cost is the first try, as it may creates a ZK connection. Another thing to do would be to limit the reload to the case it makes sense. In locateRegionInMeta there is a test on the exception:(e instanceof RegionOfflineException || e instanceof NoServerForRegionException). May be this logic could be put in ServerCallable as well, but we need to cover all cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575935#comment-13575935
]
Ted Yu commented on HBASE-6132:
---
Why commenting out assertions in TestFilterList instead of removing them ?
Patch looks good.
I ran TestJoinedScanners and it passed locally.
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-6132_94.patch, HBASE-6132_Trunk.patch,
> TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575936#comment-13575936 ] Sergey Shelukhin commented on HBASE-7667: - bq. It may be a follow on to this jira, but having "striper" dynamically add stripes at the end of the region would let allow all the stripes before the last one "go cold" which is critical for avoiding hugely wasteful compactions of non-changing data Actually, it can be added as part of the main work, HBASE-7679 (file management) code includes such capabilities. I wonder how, no matter the compactions, does region management work for such scenario. Wouldn't all the load always be on last region if you have TS keys? Or, if you have artificial partitioning but query by TS, wouldn't all queries go to all servers? bq. To major compact a stripe, all L0 files, if any, can be split into stripes, then merge all files belonging to the stripe. Can you explain more about the delete marker limitation? Suppose in current compaction selection, I choose a set of files starting at the oldest file but not including all files. Wouldn't that be enough to process delete markers that delete the updates within those files? Granted, I might not process all delete markers, but I don't have to see all files. E.g. if I only have 3 files with one entry for K each, "K=V", "delete K", "K=V2", and I compact the first two, I can remove entries for K from them, right? bq. 1. Fixed configs : in the same way that we got a lot of stability by limiting the regions/server to a fixed number, we might want to similarly limit the number of stripes per region to 10 (or X) instead of "every Y bytes". This will help us understand the benefit we get from striping and it's easy to double the striping and chart the difference. That is the original idea. Thanks for other comments :) > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a smal
[jira] [Updated] (HBASE-7815) Too subtle behavior for HConnection#getRegionLocation reload parameter and performance risk
[ https://issues.apache.org/jira/browse/HBASE-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7815: -- Summary: Too subtle behavior for HConnection#getRegionLocation reload parameter and performance risk (was: Too subtile behavior for HConnection#getRegionLocation reload parameter and performance risk) > Too subtle behavior for HConnection#getRegionLocation reload parameter and > performance risk > --- > > Key: HBASE-7815 > URL: https://issues.apache.org/jira/browse/HBASE-7815 > Project: HBase > Issue Type: Bug > Components: Client, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Minor > > HConnection#getRegionLocation(table, row, reload=true) and > HConnection#getRegionLocation(table, row, reload=false) are not equivalent > when the cache is empty: the first will check the table status while the > second will not. > As a consequence, the client won't have the same exception if the table is > disabled. With reload==true, we will have a DoNotRetryIOException, with a > message saying that the table is disabled. With reload==false we will have a > NotServingException. It's quite difficult to guess, as it's not mentioned in > the javadoc. > Second effect is that the client is going to ZooKeeper to check this table > state. In ServerCallable, if the first try is not successful, we will then go > all the time to ZK to check this status. So if a region server stops, all its > clients will connect to ZK, possibly multiple time if the recovery takes some > time. With a few hundreds clients, it's not very nice... > I'm not sure of the solution. A possible improvement in ServerCallable would > be to do a reload only at the first retry instead of all of them, but: > - it's not without side effects, even if it's limited > - the real cost is the first try, as it may creates a ZK connection. > Another thing to do would be to limit the reload to the case it makes sense. > In locateRegionInMeta there is a test on the exception:(e instanceof > RegionOfflineException || e instanceof NoServerForRegionException). > May be this logic could be put in ServerCallable as well, but we need to > cover all cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5862) After Region Close remove the Operation Metrics.
[ https://issues.apache.org/jira/browse/HBASE-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575941#comment-13575941 ] Tianying Chang commented on HBASE-5862: --- Elliott, I was working on adding some other region level metrics. I think your work around of clearing all dynamic metrics for all the regions when one region is closed works for most of the cases, but for metrics of "numericPersistentMetrics", seems not appropriate. Because clearing the metrics will lose the accumulated values which can never be recovered, especially for the live regions that are not to be closed. I want to open a bug for addressing the numericPersistentMetrics metrics, and would like to hear comments from you. Thanks > After Region Close remove the Operation Metrics. > > > Key: HBASE-5862 > URL: https://issues.apache.org/jira/browse/HBASE-5862 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.94.0, 0.96.0 > > Attachments: HBASE-5862-0.patch, HBASE-5862-1.patch, > HBASE-5862-2.patch, HBASE-5862-3.patch, HBASE-5862-4.patch, > HBASE-5862-94-3.patch, TSD.png > > > If a region is closed then Hadoop metrics shouldn't still be reporting about > that region. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7815) Too subtle behavior for HConnection#getRegionLocation reload parameter and performance risk
[ https://issues.apache.org/jira/browse/HBASE-7815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575944#comment-13575944 ] nkeywal commented on HBASE-7815: Side note: doing a "get" on a disabled table will send back DoNotRetryIOException only at the second try. The first one will be using the cache in ServerCallable, so the table state won't be checked. > Too subtle behavior for HConnection#getRegionLocation reload parameter and > performance risk > --- > > Key: HBASE-7815 > URL: https://issues.apache.org/jira/browse/HBASE-7815 > Project: HBase > Issue Type: Bug > Components: Client, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Priority: Minor > > HConnection#getRegionLocation(table, row, reload=true) and > HConnection#getRegionLocation(table, row, reload=false) are not equivalent > when the cache is empty: the first will check the table status while the > second will not. > As a consequence, the client won't have the same exception if the table is > disabled. With reload==true, we will have a DoNotRetryIOException, with a > message saying that the table is disabled. With reload==false we will have a > NotServingException. It's quite difficult to guess, as it's not mentioned in > the javadoc. > Second effect is that the client is going to ZooKeeper to check this table > state. In ServerCallable, if the first try is not successful, we will then go > all the time to ZK to check this status. So if a region server stops, all its > clients will connect to ZK, possibly multiple time if the recovery takes some > time. With a few hundreds clients, it's not very nice... > I'm not sure of the solution. A possible improvement in ServerCallable would > be to do a reload only at the first retry instead of all of them, but: > - it's not without side effects, even if it's limited > - the real cost is the first try, as it may creates a ZK connection. > Another thing to do would be to limit the reload to the case it makes sense. > In locateRegionInMeta there is a test on the exception:(e instanceof > RegionOfflineException || e instanceof NoServerForRegionException). > May be this logic could be put in ServerCallable as well, but we need to > cover all cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7816) numericPersistentMetrics should be not be cleared for regions that are not being closed.
Tianying Chang created HBASE-7816: - Summary: numericPersistentMetrics should be not be cleared for regions that are not being closed. Key: HBASE-7816 URL: https://issues.apache.org/jira/browse/HBASE-7816 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.94.4 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.94.4 when a region is closed, the region level dynamic metrics "numericPersistentMetrics" are cleared for all regions on the same region servers. It is OK for numericMetrics and timeVaryingMetrics. But not right for numericPersistentMetrics, because those value are accumulated and not be reset at poll time. To keep the right value, only the metrics for the closed region should be cleared, numericPersistentMetrics for other regions should be kept. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7816) numericPersistentMetrics should be not be cleared for regions that are not being closed.
[ https://issues.apache.org/jira/browse/HBASE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-7816: -- Priority: Minor (was: Major) > numericPersistentMetrics should be not be cleared for regions that are not > being closed. > - > > Key: HBASE-7816 > URL: https://issues.apache.org/jira/browse/HBASE-7816 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.4 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 0.94.4 > > > when a region is closed, the region level dynamic metrics > "numericPersistentMetrics" are cleared for all regions on the same region > servers. It is OK for numericMetrics and timeVaryingMetrics. But not right > for numericPersistentMetrics, because those value are accumulated and not be > reset at poll time. To keep the right value, only the metrics for the closed > region should be cleared, numericPersistentMetrics for other regions should > be kept. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7816) numericPersistentMetrics should not be cleared for regions that are not being closed
[ https://issues.apache.org/jira/browse/HBASE-7816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7816: -- Fix Version/s: (was: 0.94.4) 0.94.6 Summary: numericPersistentMetrics should not be cleared for regions that are not being closed (was: numericPersistentMetrics should be not be cleared for regions that are not being closed. ) > numericPersistentMetrics should not be cleared for regions that are not being > closed > > > Key: HBASE-7816 > URL: https://issues.apache.org/jira/browse/HBASE-7816 > Project: HBase > Issue Type: Bug > Components: metrics >Affects Versions: 0.94.4 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Fix For: 0.94.6 > > > when a region is closed, the region level dynamic metrics > "numericPersistentMetrics" are cleared for all regions on the same region > servers. It is OK for numericMetrics and timeVaryingMetrics. But not right > for numericPersistentMetrics, because those value are accumulated and not be > reset at poll time. To keep the right value, only the metrics for the closed > region should be cleared, numericPersistentMetrics for other regions should > be kept. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575951#comment-13575951 ] stack commented on HBASE-7667: -- In a striped region, if stripes >= 2 and the key distribution is basically even, a split could be done w/o references, halfhfiles, and rewriting from parent to daughter; a split could just rename the parent files into the daughter regions. It could make for split simplification and possibly make for some i/o savings. This is a load of great stuff in this issue. Best read I've had in a long time. > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[ https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7521: -- Status: Open (was: Patch Available) > fix HBASE-6060 (regions stuck in opening state) in 0.94 > --- > > Key: HBASE-7521 > URL: https://issues.apache.org/jira/browse/HBASE-7521 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: HBASE-7521-original-patch-ported-v0.patch, > HBASE-7521-original-patch-ported-v1.patch, > HBASE-7521-original-patch-ported-v2.patch, > HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch, > HBASE-7521-v1.patch > > > Discussion in HBASE-6060 implies that the fix there does not work on 0.94. > Still, we may want to fix the issue in 0.94 (via some different fix) because > the regions stuck in opening for ridiculous amounts of time is not a good > thing to have. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[ https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575960#comment-13575960 ] Sergey Shelukhin commented on HBASE-7521: - bq. This new change has been added because when in OFFLINE state we don't populate the servername right? Yes. bq. Can we throw the exception instead of catching it as in openRegion()? Seems like a normal case (multiple close-s, or close during open)? > fix HBASE-6060 (regions stuck in opening state) in 0.94 > --- > > Key: HBASE-7521 > URL: https://issues.apache.org/jira/browse/HBASE-7521 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: HBASE-7521-original-patch-ported-v0.patch, > HBASE-7521-original-patch-ported-v1.patch, > HBASE-7521-original-patch-ported-v2.patch, > HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch, > HBASE-7521-v1.patch > > > Discussion in HBASE-6060 implies that the fix there does not work on 0.94. > Still, we may want to fix the issue in 0.94 (via some different fix) because > the regions stuck in opening for ridiculous amounts of time is not a good > thing to have. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[ https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7521: -- Attachment: 7521-original-patch-ported-v4.patch Patch v4 rebases on latest 0.94 Addresses Ram's comment about duplicate message. > fix HBASE-6060 (regions stuck in opening state) in 0.94 > --- > > Key: HBASE-7521 > URL: https://issues.apache.org/jira/browse/HBASE-7521 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: 7521-original-patch-ported-v4.patch, > HBASE-7521-original-patch-ported-v0.patch, > HBASE-7521-original-patch-ported-v1.patch, > HBASE-7521-original-patch-ported-v2.patch, > HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch, > HBASE-7521-v1.patch > > > Discussion in HBASE-6060 implies that the fix there does not work on 0.94. > Still, we may want to fix the issue in 0.94 (via some different fix) because > the regions stuck in opening for ridiculous amounts of time is not a good > thing to have. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
Ian Varley created HBASE-7817: - Summary: Suggested JDWP debug options in hbase-env.sh are wrong Key: HBASE-7817 URL: https://issues.apache.org/jira/browse/HBASE-7817 Project: HBase Issue Type: Bug Reporter: Ian Varley Priority: Minor In the default hbase-env.sh in trunk, there's a section for lines you can uncomment to enable JDWP remote debugging: # Enable remote JDWP debugging of major HBase processes. Meant for Core Developers # export HBASE_MASTER_OPTS=" -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070" # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071" # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072" # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073" However, this is wrong (at least, if you're starting from local source), because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh more than once, which has the effect of including these options twice, which makes the JVM barf on startup, saying "ERROR: Cannot load this JVM TI agent twice, check your java command line for duplicate jdwp options. Error occurred during initialization of VM; agent library failed to init: jdwp". Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves the problem. This should either be changed in the default or these scripts shouldn't source hbase-env.sh multiple times I think. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7799) reassigning region stuck in open still may not work correctly due to leftover ZK node
[
https://issues.apache.org/jira/browse/HBASE-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-7799:
Attachment:
org.apache.hadoop.hbase.IntegrationTestRebalanceAndKillServersTargeted-output.txt.gz
Attaching log. Note that it has a buggy experimental feature which currently
makes HCM retry longer, but this should have no bearing on the problem...
> reassigning region stuck in open still may not work correctly due to leftover
> ZK node
> -
>
> Key: HBASE-7799
> URL: https://issues.apache.org/jira/browse/HBASE-7799
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
> Attachments:
> org.apache.hadoop.hbase.IntegrationTestRebalanceAndKillServersTargeted-output.txt.gz
>
>
> (logs grepped by region name, and abridged.
> META server was dead so OpenRegionHandler for the region took a while, and
> was interrupted:
> {code}
> 2013-02-08 14:35:01,555 DEBUG
> [RS_OPEN_REGION-10.11.2.92,64485,1360362800564-2]
> handler.OpenRegionHandler(255): Interrupting thread
> Thread[PostOpenDeployTasks:871d1c3bdf98a2c93b527cb6cc61327d,5,main]
> {code}
> Then master tried to force region offline and reassign:
> {code}
> 2013-02-08 14:35:06,500 INFO
> [MASTER_SERVER_OPERATIONS-10.11.2.92,64483,1360362800340-1]
> master.RegionStates(347): Found opening region
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OPENING, ts=1360362901596, server=10.11.2.92,64485,1360362800564} to
> be reassigned by SSH for 10.11.2.92,64485,1360362800564
> 2013-02-08 14:35:06,500 INFO
> [MASTER_SERVER_OPERATIONS-10.11.2.92,64483,1360362800340-1]
> master.RegionStates(242): Region {NAME =>
> 'IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.',
> STARTKEY => '732c', ENDKEY => '7ff8', ENCODED =>
> 871d1c3bdf98a2c93b527cb6cc61327d,} transitioned from
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OPENING, ts=1360362901596, server=10.11.2.92,64485,1360362800564} to
> {IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=CLOSED, ts=1360362906500, server=null}
> 2013-02-08 14:35:06,505 DEBUG
> [10.11.2.92,64483,1360362800340-GeneralBulkAssigner-1]
> master.AssignmentManager(1530): Forcing OFFLINE;
> was={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=CLOSED, ts=1360362906500, server=null}
> 2013-02-08 14:35:06,506 DEBUG
> [10.11.2.92,64483,1360362800340-GeneralBulkAssigner-1]
> zookeeper.ZKAssign(176): master:64483-0x13cbbf1025d Async create of
> unassigned node for 871d1c3bdf98a2c93b527cb6cc61327d with OFFLINE state
> {code}
> But didn't delete the original ZK node?
> {code}
> 2013-02-08 14:35:06,509 WARN [main-EventThread] master.OfflineCallback(59):
> Node for /hbase/region-in-transition/871d1c3bdf98a2c93b527cb6cc61327d already
> exists
> 2013-02-08 14:35:06,509 DEBUG [main-EventThread] master.OfflineCallback(69):
> rs={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OFFLINE, ts=1360362906506, server=null},
> server=10.11.2.92,64488,1360362800651
> 2013-02-08 14:35:06,512 DEBUG [main-EventThread]
> master.OfflineCallback$ExistCallback(106):
> rs={IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> state=OFFLINE, ts=1360362906506, server=null},
> server=10.11.2.92,64488,1360362800651
> {code}
> So it went into infinite cycle of failing to assign due to this:
> {code}
> 2013-02-08 14:35:06,517 INFO [PRI IPC Server handler 7 on 64488]
> regionserver.HRegionServer(3435): Received request to open region:
> IntegrationTestRebalanceAndKillServersTargeted,732c,1360362805563.871d1c3bdf98a2c93b527cb6cc61327d.
> on 10.11.2.92,64488,1360362800651
> 2013-02-08 14:35:06,521 WARN
> [RS_OPEN_REGION-10.11.2.92,64488,1360362800651-0] zookeeper.ZKAssign(762):
> regionserver:64488-0x13cbbf1025d0004 Attempt to transition the unassigned
> node for 871d1c3bdf98a2c93b527cb6cc61327d from M_ZK_REGION_OFFLINE to
> RS_ZK_REGION_OPENING failed, the node existed but was in the state
> RS_ZK_REGION_OPENING set by the server [wrong server name redacted, see
> HBASE-7798]
> {code}
> Transitioning failed-to-open similarly fails.
> It seems like master needs to nuke ZK node unconditionally to offline?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Varley updated HBASE-7817:
--
Description:
In the default hbase-env.sh in trunk, there's a section for lines you can
uncomment to enable JDWP remote debugging:
{code:none}
# Enable remote JDWP debugging of major HBase processes. Meant for Core
Developers
# export HBASE_MASTER_OPTS=" -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
{code}
However, this is wrong (at least, if you're starting from local source),
because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh more
than once, which has the effect of including these options twice, which makes
the JVM barf on startup, saying "ERROR: Cannot load this JVM TI agent twice,
check your java command line for duplicate jdwp options. Error occurred during
initialization of VM; agent library failed to init: jdwp". Removing the first
re-included instance of $HBASE_MASTER_OPTS (etc.) solves the problem. This
should either be changed in the default or these scripts shouldn't source
hbase-env.sh multiple times I think.
was:
In the default hbase-env.sh in trunk, there's a section for lines you can
uncomment to enable JDWP remote debugging:
# Enable remote JDWP debugging of major HBase processes. Meant for Core
Developers
# export HBASE_MASTER_OPTS=" -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
However, this is wrong (at least, if you're starting from local source),
because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh more
than once, which has the effect of including these options twice, which makes
the JVM barf on startup, saying "ERROR: Cannot load this JVM TI agent twice,
check your java command line for duplicate jdwp options. Error occurred during
initialization of VM; agent library failed to init: jdwp". Removing the first
re-included instance of $HBASE_MASTER_OPTS (etc.) solves the problem. This
should either be changed in the default or these scripts shouldn't source
hbase-env.sh multiple times I think.
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Reporter: Ian Varley
>Priority: Minor
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying "ERROR: Cannot load this JVM TI agent
> twice, check your java command line for duplicate jdwp options. Error
> occurred during initialization of VM; agent library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7305) ZK based Read/Write locks for table operations
[ https://issues.apache.org/jira/browse/HBASE-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7305: -- Attachment: 7305-v11.txt Patch v11 rebases on trunk. TestCatalogJanitor, TestAssignmentManager and TestConstraint passed locally. > ZK based Read/Write locks for table operations > -- > > Key: HBASE-7305 > URL: https://issues.apache.org/jira/browse/HBASE-7305 > Project: HBase > Issue Type: Bug > Components: Client, master, Zookeeper >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.96.0 > > Attachments: 7305-v11.txt, hbase-7305_v0.patch, hbase-7305_v10.patch, > hbase-7305_v1-based-on-curator.patch, hbase-7305_v2.patch, > hbase-7305_v4.patch, hbase-7305_v9.patch > > > This has started as forward porting of HBASE-5494 and HBASE-5991 from the > 89-fb branch to trunk, but diverged enough to have it's own issue. > The idea is to implement a zk based read/write lock per table. Master > initiated operations should get the write lock, and region operations (region > split, moving, balance?, etc) acquire a shared read lock. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[
https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575995#comment-13575995
]
Matt Corgan commented on HBASE-7667:
{quote}Open Times : I think this will be an issue, specifically on server
start. Need to be careful here.{quote}Hopefully could be mitigated by making
regions larger, like doubling region size and setting max 2 stripes/region.
Theoretically should be able to have the same overall number of files as normal
regions, or are there other factors at play?
{quote}Wouldn't all the load always be on last region if you have TS keys? Or,
if you have artificial partitioning but query by TS, wouldn't all queries go to
all servers?{quote}An easy strategy for P partitions is to
* prepend a single byte to each key where prefix=hash(row)%P
* pre-split the table into P regions
* tweak the balancer to evenly spread the tail partitions for each region
* writes get sprayed evenly to all tail partitions
* a single Get query will only hit one region since you know hash(row)%P
beforehand
* you scan all P partitions using a P-way collating iterator
** so yes, scans go to all servers but presumably they are huge and would hit
lots of data anyway
** because they are huge, a client that scans the partitions concurrently will
be faster
* a big multi-Get will spray to the exact servers necessary, possibly all of
them, but like scans may be faster because done in parallel
I'm not sure what most people are doing with time series data but this seems
like a good approach to me. You basically just choose arbitrarily large P. An
MD5 prefix is essentially P=2^128 (I wouldn't recommend pre-splitting at that
granularity).
> Support stripe compaction
> -
>
> Key: HBASE-7667
> URL: https://issues.apache.org/jira/browse/HBASE-7667
> Project: HBase
> Issue Type: New Feature
> Components: Compaction
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> So I was thinking about having many regions as the way to make compactions
> more manageable, and writing the level db doc about how level db range
> overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy,
> Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication
> factor.
> And I suggest the following idea, let's call it stripe compactions. It's a
> mix between level db ideas and having many small regions.
> It allows us to have a subset of benefits of many regions (wrt reads and
> compactions) without many of the drawbacks (managing and current
> memstore/etc. limitation).
> It also doesn't break seqNum-based file sorting for any one key.
> It works like this.
> The region key space is separated into configurable number of fixed-boundary
> stripes (determined the first time we stripe the data, see below).
> All the data from memstores is written to normal files with all keys present
> (not striped), similar to L0 in LevelDb, or current files.
> Compaction policy does 3 types of compactions.
> First is L0 compaction, which takes all L0 files and breaks them down by
> stripe. It may be optimized by adding more small files from different
> stripes, but the main logical outcome is that there are no more L0 files and
> all data is striped.
> Second is exactly similar to current compaction, but compacting one single
> stripe. In future, nothing prevents us from applying compaction rules and
> compacting part of the stripe (e.g. similar to current policy with rations
> and stuff, tiers, whatever), but for the first cut I'd argue let it "major
> compact" the entire stripe. Or just have the ratio and no more complexity.
> Finally, the third addresses the concern of the fixed boundaries causing
> stripes to be very unbalanced.
> It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the
> results out with different boundaries.
> There's a tradeoff here - if we always take 2 adjacent stripes, compactions
> will be smaller but rebalancing will take ridiculous amount of I/O.
> If we take many stripes we are essentially getting into the
> epic-major-compaction problem again. Some heuristics will have to be in place.
> In general, if, before stripes are determined, we initially let L0 grow
> before determining the stripes, we will get better boundaries.
> Also, unless unbalancing is really large we don't need to rebalance really.
> Obviously this scheme (as well as level) is not applicable for all scenarios,
> e.g. if timestamp is your key it completely falls apart.
> The end result:
> - many small compactions that can be spread out in time.
> - reads still read from a small number of files (one stripe + L0).
> - region splits become marvelously simple (if we could move files between
> regions, no references would be needed).
> Main adv
[jira] [Updated] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-7817:
--
Description:
In the default hbase-env.sh in trunk, there's a section for lines you can
uncomment to enable JDWP remote debugging:
{code:none}
# Enable remote JDWP debugging of major HBase processes. Meant for Core
Developers
# export HBASE_MASTER_OPTS=" -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
{code}
However, this is wrong (at least, if you're starting from local source),
because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh more
than once, which has the effect of including these options twice, which makes
the JVM barf on startup, saying
"ERROR: Cannot load this JVM TI agent twice, check your java command line for
duplicate jdwp options. Error occurred during initialization of VM; agent
library failed to init: jdwp".
Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves the
problem. This should either be changed in the default or these scripts
shouldn't source hbase-env.sh multiple times I think.
was:
In the default hbase-env.sh in trunk, there's a section for lines you can
uncomment to enable JDWP remote debugging:
{code:none}
# Enable remote JDWP debugging of major HBase processes. Meant for Core
Developers
# export HBASE_MASTER_OPTS=" -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
# export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
# export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
{code}
However, this is wrong (at least, if you're starting from local source),
because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh more
than once, which has the effect of including these options twice, which makes
the JVM barf on startup, saying "ERROR: Cannot load this JVM TI agent twice,
check your java command line for duplicate jdwp options. Error occurred during
initialization of VM; agent library failed to init: jdwp". Removing the first
re-included instance of $HBASE_MASTER_OPTS (etc.) solves the problem. This
should either be changed in the default or these scripts shouldn't source
hbase-env.sh multiple times I think.
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Reporter: Ian Varley
>Priority: Minor
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7817) Suggested JDWP debug options in hbase-env.sh are wrong
[
https://issues.apache.org/jira/browse/HBASE-7817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576002#comment-13576002
]
Ted Yu commented on HBASE-7817:
---
Or change the JDWP enabling comments in hbase-env.sh so that we detect whether
JDWP is already enabled in $HBASE_MASTER_OPTS, etc.
> Suggested JDWP debug options in hbase-env.sh are wrong
> --
>
> Key: HBASE-7817
> URL: https://issues.apache.org/jira/browse/HBASE-7817
> Project: HBase
> Issue Type: Bug
>Reporter: Ian Varley
>Priority: Minor
>
> In the default hbase-env.sh in trunk, there's a section for lines you can
> uncomment to enable JDWP remote debugging:
> {code:none}
> # Enable remote JDWP debugging of major HBase processes. Meant for Core
> Developers
> # export HBASE_MASTER_OPTS=" -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8070"
> # export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8071"
> # export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8072"
> # export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8073"
> {code}
> However, this is wrong (at least, if you're starting from local source),
> because somewhere in the chain of start-hbase.sh, it sources hbase-env.sh
> more than once, which has the effect of including these options twice, which
> makes the JVM barf on startup, saying
> "ERROR: Cannot load this JVM TI agent twice, check your java command line for
> duplicate jdwp options. Error occurred during initialization of VM; agent
> library failed to init: jdwp".
> Removing the first re-included instance of $HBASE_MASTER_OPTS (etc.) solves
> the problem. This should either be changed in the default or these scripts
> shouldn't source hbase-env.sh multiple times I think.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7804) Minor typo in Appendix B. hbck In Depth section of book
[
https://issues.apache.org/jira/browse/HBASE-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576012#comment-13576012
]
Jonathan Hsieh commented on HBASE-7804:
---
Nice catch [~rajesh23], I've committed to trunk. Thanks!
> Minor typo in Appendix B. hbck In Depth section of book
> ---
>
> Key: HBASE-7804
> URL: https://issues.apache.org/jira/browse/HBASE-7804
> Project: HBase
> Issue Type: Bug
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Attachments: HBASE-7804.patch
>
>
> last slash at ./bin/hbase/ is not required.
> {code}
> $ ./bin/hbase/ hbck TableFoo TableBar
> {code}
> {code}
> $ ./bin/hbase/ hbck -repair TableFoo TableBar
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7818) add region level metrics readReqeustCount and writeRequestCount
Tianying Chang created HBASE-7818: - Summary: add region level metrics readReqeustCount and writeRequestCount Key: HBASE-7818 URL: https://issues.apache.org/jira/browse/HBASE-7818 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.94.4 Reporter: Tianying Chang Assignee: Tianying Chang Priority: Minor Fix For: 0.94.6 Request rate at region server level can help identify the hot region server. But it will be good if we can further identify the hot regions on that region server. That way, we can easily find out unbalanced regions problem. Currently, readRequestCount and writeReqeustCount per region is exposed at webUI. It will be more useful to expose it through hadoop metrics framework and/or JMX, so that people can see the history when the region is hot. I am exposing the existing readRequestCount/writeRequestCount into the dynamic region level metrics framework. I am not changing/exposing it as rate because our openTSDB is taking the raw data of read/write count, and apply rate function to display the rate already. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7804) Minor typo in Appendix B. hbck In Depth section of book
[
https://issues.apache.org/jira/browse/HBASE-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7804:
--
Labels: docs (was: )
> Minor typo in Appendix B. hbck In Depth section of book
> ---
>
> Key: HBASE-7804
> URL: https://issues.apache.org/jira/browse/HBASE-7804
> Project: HBase
> Issue Type: Bug
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Labels: docs
> Fix For: 0.96.0
>
> Attachments: HBASE-7804.patch
>
>
> last slash at ./bin/hbase/ is not required.
> {code}
> $ ./bin/hbase/ hbck TableFoo TableBar
> {code}
> {code}
> $ ./bin/hbase/ hbck -repair TableFoo TableBar
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7804) Minor typo in Appendix B. hbck In Depth section of book
[
https://issues.apache.org/jira/browse/HBASE-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7804:
--
Resolution: Fixed
Fix Version/s: 0.96.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Minor typo in Appendix B. hbck In Depth section of book
> ---
>
> Key: HBASE-7804
> URL: https://issues.apache.org/jira/browse/HBASE-7804
> Project: HBase
> Issue Type: Bug
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Fix For: 0.96.0
>
> Attachments: HBASE-7804.patch
>
>
> last slash at ./bin/hbase/ is not required.
> {code}
> $ ./bin/hbase/ hbck TableFoo TableBar
> {code}
> {code}
> $ ./bin/hbase/ hbck -repair TableFoo TableBar
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HBASE-7814: --- Assignee: Sergey Shelukhin > Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 > > > Key: HBASE-7814 > URL: https://issues.apache.org/jira/browse/HBASE-7814 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7804) Minor typo in Appendix B. hbck In Depth section of book
[
https://issues.apache.org/jira/browse/HBASE-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7804:
--
Component/s: hbck
> Minor typo in Appendix B. hbck In Depth section of book
> ---
>
> Key: HBASE-7804
> URL: https://issues.apache.org/jira/browse/HBASE-7804
> Project: HBase
> Issue Type: Bug
> Components: documentation, hbck
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Labels: docs
> Fix For: 0.96.0
>
> Attachments: HBASE-7804.patch
>
>
> last slash at ./bin/hbase/ is not required.
> {code}
> $ ./bin/hbase/ hbck TableFoo TableBar
> {code}
> {code}
> $ ./bin/hbase/ hbck -repair TableFoo TableBar
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7804) Minor typo in Appendix B. hbck In Depth section of book
[
https://issues.apache.org/jira/browse/HBASE-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-7804:
--
Component/s: documentation
> Minor typo in Appendix B. hbck In Depth section of book
> ---
>
> Key: HBASE-7804
> URL: https://issues.apache.org/jira/browse/HBASE-7804
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Labels: docs
> Fix For: 0.96.0
>
> Attachments: HBASE-7804.patch
>
>
> last slash at ./bin/hbase/ is not required.
> {code}
> $ ./bin/hbase/ hbck TableFoo TableBar
> {code}
> {code}
> $ ./bin/hbase/ hbck -repair TableFoo TableBar
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-6132:
--
Attachment: 6132-trunk-v2.txt
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 6132-trunk-v2.txt, HBASE-6132_94.patch,
> HBASE-6132_Trunk.patch, TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7761) MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
[ https://issues.apache.org/jira/browse/HBASE-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7761: -- Attachment: 7761_Trunk.patch > MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true > > > Key: HBASE-7761 > URL: https://issues.apache.org/jira/browse/HBASE-7761 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2, 0.96.0, 0.94.4 >Reporter: Jean-Daniel Cryans >Priority: Minor > Fix For: 0.96.0, 0.94.6 > > Attachments: 7761_Trunk.patch, HBASE-7761_94.patch, > HBASE-7761_Trunk.patch > > > Someone not familiar with the MemStore code might find it confusing that > MemStore.USEMSLAB_DEFAULT's value is "false" whereas > hbase.hregion.memstore.mslab.enabled is true in hbase-default.xml (and in the > end hbase-default.xml is the one used). > This should be harmonized to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7761) MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
[ https://issues.apache.org/jira/browse/HBASE-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7761: -- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true > > > Key: HBASE-7761 > URL: https://issues.apache.org/jira/browse/HBASE-7761 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.4, 0.92.2, 0.96.0 >Reporter: Jean-Daniel Cryans >Priority: Minor > Fix For: 0.96.0, 0.94.6 > > Attachments: 7761_Trunk.patch, HBASE-7761_94.patch, > HBASE-7761_Trunk.patch > > > Someone not familiar with the MemStore code might find it confusing that > MemStore.USEMSLAB_DEFAULT's value is "false" whereas > hbase.hregion.memstore.mslab.enabled is true in hbase-default.xml (and in the > end hbase-default.xml is the one used). > This should be harmonized to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7761) MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
[ https://issues.apache.org/jira/browse/HBASE-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-7761: - Assignee: Anoop Sam John > MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true > > > Key: HBASE-7761 > URL: https://issues.apache.org/jira/browse/HBASE-7761 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2, 0.96.0, 0.94.4 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John >Priority: Minor > Fix For: 0.96.0, 0.94.6 > > Attachments: 7761_Trunk.patch, HBASE-7761_94.patch, > HBASE-7761_Trunk.patch > > > Someone not familiar with the MemStore code might find it confusing that > MemStore.USEMSLAB_DEFAULT's value is "false" whereas > hbase.hregion.memstore.mslab.enabled is true in hbase-default.xml (and in the > end hbase-default.xml is the one used). > This should be harmonized to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576023#comment-13576023 ] Sergey Shelukhin commented on HBASE-7667: - bq. a split could just rename the parent files into the daughter regions I was told this is impossible due to snapshots relying on files not moving between regions (or on references during splits?) We just discussed this here, some improvements for splits should definitely be possible. bq. so yes, scans go to all servers but presumably they are huge and would hit lots of data anyway Yeah, I meant the scans, was assuming scans for TS data mostly. I see. > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7819) consider split improvements for stripe compactions
Sergey Shelukhin created HBASE-7819: --- Summary: consider split improvements for stripe compactions Key: HBASE-7819 URL: https://issues.apache.org/jira/browse/HBASE-7819 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin >From discussions in HBASE-7667, and in person. Files cannot be "just moved" to new regions because of snapshot reliance on that not happening (correct me if I'm wrong). However, obviously we don't need compaction after split if we have files with specific ranges, so we could handle this somehow. On the other hand, if you have 6 fixed stripes, and split into 3 and 3, you'd want to go back to 6 stripes, so when you do compaction in one of the 3 you will want special handling to do it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576044#comment-13576044 ] Jimmy Xiang commented on HBASE-7667: bq. Wouldn't that be enough to process delete markers that delete the updates within those files? I think so, as long as the files not included are newer (like in L0). > Support stripe compaction > - > > Key: HBASE-7667 > URL: https://issues.apache.org/jira/browse/HBASE-7667 > Project: HBase > Issue Type: New Feature > Components: Compaction >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > So I was thinking about having many regions as the way to make compactions > more manageable, and writing the level db doc about how level db range > overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, > Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication > factor. > And I suggest the following idea, let's call it stripe compactions. It's a > mix between level db ideas and having many small regions. > It allows us to have a subset of benefits of many regions (wrt reads and > compactions) without many of the drawbacks (managing and current > memstore/etc. limitation). > It also doesn't break seqNum-based file sorting for any one key. > It works like this. > The region key space is separated into configurable number of fixed-boundary > stripes (determined the first time we stripe the data, see below). > All the data from memstores is written to normal files with all keys present > (not striped), similar to L0 in LevelDb, or current files. > Compaction policy does 3 types of compactions. > First is L0 compaction, which takes all L0 files and breaks them down by > stripe. It may be optimized by adding more small files from different > stripes, but the main logical outcome is that there are no more L0 files and > all data is striped. > Second is exactly similar to current compaction, but compacting one single > stripe. In future, nothing prevents us from applying compaction rules and > compacting part of the stripe (e.g. similar to current policy with rations > and stuff, tiers, whatever), but for the first cut I'd argue let it "major > compact" the entire stripe. Or just have the ratio and no more complexity. > Finally, the third addresses the concern of the fixed boundaries causing > stripes to be very unbalanced. > It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the > results out with different boundaries. > There's a tradeoff here - if we always take 2 adjacent stripes, compactions > will be smaller but rebalancing will take ridiculous amount of I/O. > If we take many stripes we are essentially getting into the > epic-major-compaction problem again. Some heuristics will have to be in place. > In general, if, before stripes are determined, we initially let L0 grow > before determining the stripes, we will get better boundaries. > Also, unless unbalancing is really large we don't need to rebalance really. > Obviously this scheme (as well as level) is not applicable for all scenarios, > e.g. if timestamp is your key it completely falls apart. > The end result: > - many small compactions that can be spread out in time. > - reads still read from a small number of files (one stripe + L0). > - region splits become marvelously simple (if we could move files between > regions, no references would be needed). > Main advantage over Level (for HBase) is that default store can still open > the files and get correct results - there are no range overlap shenanigans. > It also needs no metadata, although we may record some for convenience. > It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576045#comment-13576045 ] Elliott Clark commented on HBASE-3171: -- Minor stuff to remove: * Can remove the constants for ROOT from HConstants. * Root's HRegionInfo can be removed. * Root's HTableDescriptor can be removed. * MetaMigrationConvertingToPB still has a bunch of Root related stuff. * The region list template still explains what root is for. Getting into the code now. Looks good. > Drop ROOT and instead store META location(s) directly in ZooKeeper > -- > > Key: HBASE-3171 > URL: https://issues.apache.org/jira/browse/HBASE-3171 > Project: HBase > Issue Type: Improvement > Components: Client, master, regionserver, Zookeeper >Reporter: Jonathan Gray > Attachments: HBASE-3171.patch, HBASE-3171-v2.patch, > HBASE-3171-v3.patch, HBASE-3171-v4.patch > > > Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and > reading the META location, we should just store the META location directly in > ZooKeeper. > The purpose of the root region from the bigtable paper was to support > multiple meta regions. Currently, we explicitly only support a single meta > region, so the translation from our current code of a single root location to > a single meta location will be very simple. Long-term, it seems reasonable > that we could store several meta region locations in ZK. There's been some > discussion in HBASE-1755 about actually moving META into ZK, but I think this > jira is a good step towards taking some of the complexity out of how we have > to deal with catalog tables everywhere. > As-is, a new client already requires ZK to get the root location, so this > would not change those requirements in any way. > The primary motivation for this is to simplify things like CatalogTracker. > The way we can handle root in that class is really simple but the tracking of > meta is difficulty and a bit hacky. This hack on tracking of the meta > location is what caused one of the bugs over in HBASE-3159. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[ https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7521: Attachment: HBASE-7521-v5.patch fixed test issue > fix HBASE-6060 (regions stuck in opening state) in 0.94 > --- > > Key: HBASE-7521 > URL: https://issues.apache.org/jira/browse/HBASE-7521 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: 7521-original-patch-ported-v4.patch, > HBASE-7521-original-patch-ported-v0.patch, > HBASE-7521-original-patch-ported-v1.patch, > HBASE-7521-original-patch-ported-v2.patch, > HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch, > HBASE-7521-v1.patch, HBASE-7521-v5.patch > > > Discussion in HBASE-6060 implies that the fix there does not work on 0.94. > Still, we may want to fix the issue in 0.94 (via some different fix) because > the regions stuck in opening for ridiculous amounts of time is not a good > thing to have. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7814: Attachment: HBASE-7814-v0.patch Automatic port; I am running tests > Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 > > > Key: HBASE-7814 > URL: https://issues.apache.org/jira/browse/HBASE-7814 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: HBASE-7814-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7814: Status: Patch Available (was: Open) > Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 > > > Key: HBASE-7814 > URL: https://issues.apache.org/jira/browse/HBASE-7814 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: HBASE-7814-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7305) ZK based Read/Write locks for table operations
[
https://issues.apache.org/jira/browse/HBASE-7305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576070#comment-13576070
]
Hadoop QA commented on HBASE-7305:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568865/7305-v11.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.constraint.TestConstraint
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4413//console
This message is automatically generated.
> ZK based Read/Write locks for table operations
> --
>
> Key: HBASE-7305
> URL: https://issues.apache.org/jira/browse/HBASE-7305
> Project: HBase
> Issue Type: Bug
> Components: Client, master, Zookeeper
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.96.0
>
> Attachments: 7305-v11.txt, hbase-7305_v0.patch, hbase-7305_v10.patch,
> hbase-7305_v1-based-on-curator.patch, hbase-7305_v2.patch,
> hbase-7305_v4.patch, hbase-7305_v9.patch
>
>
> This has started as forward porting of HBASE-5494 and HBASE-5991 from the
> 89-fb branch to trunk, but diverged enough to have it's own issue.
> The idea is to implement a zk based read/write lock per table. Master
> initiated operations should get the write lock, and region operations (region
> split, moving, balance?, etc) acquire a shared read lock.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576073#comment-13576073
]
Hadoop QA commented on HBASE-7814:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568876/HBASE-7814-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4416//console
This message is automatically generated.
> Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
>
>
> Key: HBASE-7814
> URL: https://issues.apache.org/jira/browse/HBASE-7814
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
> Fix For: 0.94.6
>
> Attachments: HBASE-7814-v0.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[
https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576083#comment-13576083
]
Jimmy Xiang commented on HBASE-7521:
@Ram, sure. I am ok with getting this to 0.94. I was just in doubt this will
fix all racing issues. For example, the following code (in #assign) is
synchronized on the regionstate. However, AM#handleRegion is synchronized on
regionsInTransition. Suppose serverManager.sendRegionOpen returns
RegionOpeningState.OPENED, but OpenRegionHandler fails to open it and
transitions the ZK node to RS_ZK_REGION_FAILED_OPEN, then handleRegion will
transition the region state to Closed, right after that, this thread
transitions the state back to PENDING_OPEN, which may cause some confusion
further down the road.
{code}
+RegionOpeningState regionOpenState =
serverManager.sendRegionOpen(plan.getDestination(),
+state.getRegion(), versionOfOfflineNode);
+if (regionOpenState == RegionOpeningState.OPENED) {
+ // Transition RegionState to PENDING_OPEN
+ if (state.isOffline() && !state.isOpening()) {
+state.update(RegionState.State.PENDING_OPEN,
+System.currentTimeMillis(), plan.getDestination());
+ }
{code}
> fix HBASE-6060 (regions stuck in opening state) in 0.94
> ---
>
> Key: HBASE-7521
> URL: https://issues.apache.org/jira/browse/HBASE-7521
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.94.6
>
> Attachments: 7521-original-patch-ported-v4.patch,
> HBASE-7521-original-patch-ported-v0.patch,
> HBASE-7521-original-patch-ported-v1.patch,
> HBASE-7521-original-patch-ported-v2.patch,
> HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch,
> HBASE-7521-v1.patch, HBASE-7521-v5.patch
>
>
> Discussion in HBASE-6060 implies that the fix there does not work on 0.94.
> Still, we may want to fix the issue in 0.94 (via some different fix) because
> the regions stuck in opening for ridiculous amounts of time is not a good
> thing to have.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7761) MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
[
https://issues.apache.org/jira/browse/HBASE-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576091#comment-13576091
]
Hadoop QA commented on HBASE-7761:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568874/7761_Trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4414//console
This message is automatically generated.
> MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
>
>
> Key: HBASE-7761
> URL: https://issues.apache.org/jira/browse/HBASE-7761
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.2, 0.96.0, 0.94.4
>Reporter: Jean-Daniel Cryans
>Assignee: Anoop Sam John
>Priority: Minor
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 7761_Trunk.patch, HBASE-7761_94.patch,
> HBASE-7761_Trunk.patch
>
>
> Someone not familiar with the MemStore code might find it confusing that
> MemStore.USEMSLAB_DEFAULT's value is "false" whereas
> hbase.hregion.memstore.mslab.enabled is true in hbase-default.xml (and in the
> end hbase-default.xml is the one used).
> This should be harmonized to true.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7761) MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
[ https://issues.apache.org/jira/browse/HBASE-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576095#comment-13576095 ] Ted Yu commented on HBASE-7761: --- Integrated to 0.94 and trunk. Thanks for the patch, Anoop. Thanks for the review, J-D. > MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true > > > Key: HBASE-7761 > URL: https://issues.apache.org/jira/browse/HBASE-7761 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2, 0.96.0, 0.94.4 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John >Priority: Minor > Fix For: 0.96.0, 0.94.6 > > Attachments: 7761_Trunk.patch, HBASE-7761_94.patch, > HBASE-7761_Trunk.patch > > > Someone not familiar with the MemStore code might find it confusing that > MemStore.USEMSLAB_DEFAULT's value is "false" whereas > hbase.hregion.memstore.mslab.enabled is true in hbase-default.xml (and in the > end hbase-default.xml is the one used). > This should be harmonized to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576096#comment-13576096
]
Jean-Daniel Cryans commented on HBASE-3171:
---
Thanks for looking at the patch guys, much appreciated.
bq. Why not remove the first if block ?
I'm still on the fence about removing some of the core code. We may need to
keep it around for migration/upgrade, which I haven't tackled yet.
bq. It would be better if the above method can be removed. It was created to
accommodate both -ROOT- and .META. tables.
Comment above + I'd like to verify the semantics of meta tables. It's probably
fine to remove, but not there yet.
bq. Should the above method belong to ServerName class ?
I'll look at it, as you can see this new class just has the code from
RootRegionTracker.
{quote}
Can remove the constants for ROOT from HConstants.
etc
{quote}
See my first answer to Ted.
bq. The region list template still explains what root is for.
Ah I missed it in my rebase I think, thanks!
> Drop ROOT and instead store META location(s) directly in ZooKeeper
> --
>
> Key: HBASE-3171
> URL: https://issues.apache.org/jira/browse/HBASE-3171
> Project: HBase
> Issue Type: Improvement
> Components: Client, master, regionserver, Zookeeper
>Reporter: Jonathan Gray
> Attachments: HBASE-3171.patch, HBASE-3171-v2.patch,
> HBASE-3171-v3.patch, HBASE-3171-v4.patch
>
>
> Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and
> reading the META location, we should just store the META location directly in
> ZooKeeper.
> The purpose of the root region from the bigtable paper was to support
> multiple meta regions. Currently, we explicitly only support a single meta
> region, so the translation from our current code of a single root location to
> a single meta location will be very simple. Long-term, it seems reasonable
> that we could store several meta region locations in ZK. There's been some
> discussion in HBASE-1755 about actually moving META into ZK, but I think this
> jira is a good step towards taking some of the complexity out of how we have
> to deal with catalog tables everywhere.
> As-is, a new client already requires ZK to get the root location, so this
> would not change those requirements in any way.
> The primary motivation for this is to simplify things like CatalogTracker.
> The way we can handle root in that class is really simple but the tracking of
> meta is difficulty and a bit hacky. This hack on tracking of the meta
> location is what caused one of the bugs over in HBASE-3159.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576107#comment-13576107 ] Jimmy Xiang commented on HBASE-7814: +1 > Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 > > > Key: HBASE-7814 > URL: https://issues.apache.org/jira/browse/HBASE-7814 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: HBASE-7814-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576111#comment-13576111
]
Hadoop QA commented on HBASE-6132:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568872/6132-trunk-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4415//console
This message is automatically generated.
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 6132-trunk-v2.txt, HBASE-6132_94.patch,
> HBASE-6132_Trunk.patch, TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7820) Authenticating users from different realm without a trust relationship
Benoy Antony created HBASE-7820: --- Summary: Authenticating users from different realm without a trust relationship Key: HBASE-7820 URL: https://issues.apache.org/jira/browse/HBASE-7820 Project: HBase Issue Type: Improvement Components: security Reporter: Benoy Antony HBase servers are part of the Hadoop domain, controlled by Hadoop Active Directory. The users belong to the CORP domain, controlled by the CORP Active Directory. In the absence of a one way trust from HADOOP DOMAIN to CORP DOMAIN, how will HBase servers authenticate CORP users ? This is the HBase equivalent of HADOOP-9296 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7820) Authenticating users from different realm without a trust relationship
[ https://issues.apache.org/jira/browse/HBASE-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HBASE-7820: Attachment: HBASE-7820-0.94.patch > Authenticating users from different realm without a trust relationship > -- > > Key: HBASE-7820 > URL: https://issues.apache.org/jira/browse/HBASE-7820 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Benoy Antony > Attachments: HBASE-7820-0.94.patch > > > HBase servers are part of the Hadoop domain, controlled by Hadoop Active > Directory. > The users belong to the CORP domain, controlled by the CORP Active Directory. > In the absence of a one way trust from HADOOP DOMAIN to CORP DOMAIN, how will > HBase servers authenticate CORP users ? > This is the HBase equivalent of HADOOP-9296 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576123#comment-13576123
]
Ted Yu commented on HBASE-6132:
---
Integrated to trunk.
Thanks for the patch, Anoop.
Thanks for the review, Ram.
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 6132-trunk-v2.txt, HBASE-6132_94.patch,
> HBASE-6132_Trunk.patch, TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7821) Remove Duplicated code in CompactSelection
Elliott Clark created HBASE-7821: Summary: Remove Duplicated code in CompactSelection Key: HBASE-7821 URL: https://issues.apache.org/jira/browse/HBASE-7821 Project: HBase Issue Type: Task Reporter: Elliott Clark Assignee: Elliott Clark Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7821) Remove Duplicated code in CompactSelection
[ https://issues.apache.org/jira/browse/HBASE-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7821: - Affects Version/s: 0.96.0 > Remove Duplicated code in CompactSelection > -- > > Key: HBASE-7821 > URL: https://issues.apache.org/jira/browse/HBASE-7821 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Minor > Attachments: HBASE-7821-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7821) Remove Duplicated code in CompactSelection
[ https://issues.apache.org/jira/browse/HBASE-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7821: - Attachment: HBASE-7821-0.patch > Remove Duplicated code in CompactSelection > -- > > Key: HBASE-7821 > URL: https://issues.apache.org/jira/browse/HBASE-7821 > Project: HBase > Issue Type: Task >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Minor > Attachments: HBASE-7821-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7821) Remove Duplicated code in CompactSelection
[ https://issues.apache.org/jira/browse/HBASE-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7821: - Status: Patch Available (was: Open) > Remove Duplicated code in CompactSelection > -- > > Key: HBASE-7821 > URL: https://issues.apache.org/jira/browse/HBASE-7821 > Project: HBase > Issue Type: Task >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Minor > Attachments: HBASE-7821-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7821) Remove Duplicated code in CompactSelection
[ https://issues.apache.org/jira/browse/HBASE-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7821: - Component/s: Compaction > Remove Duplicated code in CompactSelection > -- > > Key: HBASE-7821 > URL: https://issues.apache.org/jira/browse/HBASE-7821 > Project: HBase > Issue Type: Task > Components: Compaction >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Minor > Attachments: HBASE-7821-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7821) Remove Duplicated code in CompactSelection
[ https://issues.apache.org/jira/browse/HBASE-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576131#comment-13576131 ] stack commented on HBASE-7821: -- +1 if hadoopqa is ok w/ it. > Remove Duplicated code in CompactSelection > -- > > Key: HBASE-7821 > URL: https://issues.apache.org/jira/browse/HBASE-7821 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Minor > Attachments: HBASE-7821-0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7521) fix HBASE-6060 (regions stuck in opening state) in 0.94
[
https://issues.apache.org/jira/browse/HBASE-7521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576133#comment-13576133
]
Ted Yu commented on HBASE-7521:
---
Ran through 0.94 test suite:
{code}
Tests run: 1272, Failures: 0, Errors: 0, Skipped: 13
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 46:17.736s
[INFO] Finished at: Mon Feb 11 21:08:22 UTC 2013
{code}
Plan to get the patch in by Friday.
> fix HBASE-6060 (regions stuck in opening state) in 0.94
> ---
>
> Key: HBASE-7521
> URL: https://issues.apache.org/jira/browse/HBASE-7521
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.94.6
>
> Attachments: 7521-original-patch-ported-v4.patch,
> HBASE-7521-original-patch-ported-v0.patch,
> HBASE-7521-original-patch-ported-v1.patch,
> HBASE-7521-original-patch-ported-v2.patch,
> HBASE-7521-original-patch-ported-v3.patch, HBASE-7521-v0.patch,
> HBASE-7521-v1.patch, HBASE-7521-v5.patch
>
>
> Discussion in HBASE-6060 implies that the fix there does not work on 0.94.
> Still, we may want to fix the issue in 0.94 (via some different fix) because
> the regions stuck in opening for ridiculous amounts of time is not a good
> thing to have.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7763) Compactions not sorting based on size anymore.
[ https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576142#comment-13576142 ] stack commented on HBASE-7763: -- +1 on patch. Keeps seqid sorting as first order sort. > Compactions not sorting based on size anymore. > -- > > Key: HBASE-7763 > URL: https://issues.apache.org/jira/browse/HBASE-7763 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 0.96.0, 0.94.4 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0, 0.94.6 > > Attachments: HBASE-7763-trunk-1.patch, HBASE-7763-trunk-2.patch, > HBASE-7763-trunk-3.patch, HBASE-7763-trunk-TESTING.patch, > HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch > > > Currently compaction selection is not sorting based on size. This causes > selection to choose larger files to re-write than are needed when bulk loads > are involved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter & PageFilter not working with FilterList
[
https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576145#comment-13576145
]
Ted Yu commented on HBASE-6132:
---
Integrated to 0.94 as well.
> ColumnCountGetFilter & PageFilter not working with FilterList
> -
>
> Key: HBASE-6132
> URL: https://issues.apache.org/jira/browse/HBASE-6132
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 0.92.0, 0.92.1, 0.94.0
> Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0,
> zookeeper 3.4.3
>Reporter: Benjamin Kim
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: 6132-trunk-v2.txt, HBASE-6132_94.patch,
> HBASE-6132_Trunk.patch, TestColumnCountGetFilter.java
>
>
> Thanks to Anoop and Ramkrishna, here's what we found with FilterList
> If I use FilterList to include ColumnCountGetFilter among other filters, the
> returning Result has no keyvalues.
> This problem seems to occur when specified column count is less then actual
> number of existing columns.
> Also same problem arises with PageFilter
> Following is the code of the problem:
> {code}
> Configuration conf = HBaseConfiguration.create();
> HTable table = new HTable(conf, "test");
> Get get = new Get(Bytes.toBytes("test1"));
> FilterList filterList = new FilterList();
> filterList.addFilter(new ColumnCountGetFilter(100));
> get.setFilter(filterList);
> Result r = table.get(get);
> System.out.println(r.size()); // prints zero
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7820) Authenticating users from different realm without a trust relationship
[
https://issues.apache.org/jira/browse/HBASE-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576151#comment-13576151
]
Ted Yu commented on HBASE-7820:
---
@Benoy:
Can you attach patch for trunk ?
Looks like you forgot to include HBaseMultiRealmUserAuthentication source code.
{code}
+ } else if (MultiRealmUserAuthentication.isAUserInADifferentRealm (
+ ticket.getUGI(), conf) ){
{code}
Please leave one space between keywords. You can use
dev-support/hbase_eclipse_formatter.xml if Eclipse is your IDE.
isAUserInADifferentRealm -> isUserInADifferentRealm (only one user is passed to
this method)
{code}
+ }
+ else {
+ authMethod = AuthMethod.KERBEROS;
{code}
Move 'else' to the line above. Indentation should be two spaces.
{code}
+ if (LOG.isDebugEnabled())
+LOG.debug("Kerberos principal name is " + fullName);
{code}
Include the log in curly braces.
> Authenticating users from different realm without a trust relationship
> --
>
> Key: HBASE-7820
> URL: https://issues.apache.org/jira/browse/HBASE-7820
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Benoy Antony
> Attachments: HBASE-7820-0.94.patch
>
>
> HBase servers are part of the Hadoop domain, controlled by Hadoop Active
> Directory.
> The users belong to the CORP domain, controlled by the CORP Active Directory.
> In the absence of a one way trust from HADOOP DOMAIN to CORP DOMAIN, how will
> HBase servers authenticate CORP users ?
> This is the HBase equivalent of HADOOP-9296
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7761) MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true
[ https://issues.apache.org/jira/browse/HBASE-7761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576155#comment-13576155 ] Hudson commented on HBASE-7761: --- Integrated in HBase-TRUNK #3866 (See [https://builds.apache.org/job/HBase-TRUNK/3866/]) HBASE-7761 MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true (Anoop) (Revision 1444952) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java > MemStore.USEMSLAB_DEFAULT is false, hbase-default.xml says it's true > > > Key: HBASE-7761 > URL: https://issues.apache.org/jira/browse/HBASE-7761 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2, 0.96.0, 0.94.4 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John >Priority: Minor > Fix For: 0.96.0, 0.94.6 > > Attachments: 7761_Trunk.patch, HBASE-7761_94.patch, > HBASE-7761_Trunk.patch > > > Someone not familiar with the MemStore code might find it confusing that > MemStore.USEMSLAB_DEFAULT's value is "false" whereas > hbase.hregion.memstore.mslab.enabled is true in hbase-default.xml (and in the > end hbase-default.xml is the one used). > This should be harmonized to true. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7804) Minor typo in Appendix B. hbck In Depth section of book
[
https://issues.apache.org/jira/browse/HBASE-7804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13576154#comment-13576154
]
Hudson commented on HBASE-7804:
---
Integrated in HBase-TRUNK #3866 (See
[https://builds.apache.org/job/HBase-TRUNK/3866/])
HBASE-7804 Minor typo in Appendix B. hbck In Depth section of book
(rajeshbabu) (Revision 1444929)
Result = FAILURE
jmhsieh :
Files :
* /hbase/trunk/src/docbkx/book.xml
> Minor typo in Appendix B. hbck In Depth section of book
> ---
>
> Key: HBASE-7804
> URL: https://issues.apache.org/jira/browse/HBASE-7804
> Project: HBase
> Issue Type: Bug
> Components: documentation, hbck
>Reporter: rajeshbabu
>Assignee: rajeshbabu
>Priority: Minor
> Labels: docs
> Fix For: 0.96.0
>
> Attachments: HBASE-7804.patch
>
>
> last slash at ./bin/hbase/ is not required.
> {code}
> $ ./bin/hbase/ hbck TableFoo TableBar
> {code}
> {code}
> $ ./bin/hbase/ hbck -repair TableFoo TableBar
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7763) Compactions not sorting based on size anymore.
[ https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7763: - Resolution: Fixed Fix Version/s: (was: 0.94.6) Status: Resolved (was: Patch Available) > Compactions not sorting based on size anymore. > -- > > Key: HBASE-7763 > URL: https://issues.apache.org/jira/browse/HBASE-7763 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 0.96.0, 0.94.4 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Critical > Fix For: 0.96.0 > > Attachments: HBASE-7763-trunk-1.patch, HBASE-7763-trunk-2.patch, > HBASE-7763-trunk-3.patch, HBASE-7763-trunk-TESTING.patch, > HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch > > > Currently compaction selection is not sorting based on size. This causes > selection to choose larger files to re-write than are needed when bulk loads > are involved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7814) Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7814: -- Hadoop Flags: Reviewed Integrated to 0.94 Thanks for the patch, Sergey. Thanks for the review, Jimmy. > Port HBASE-6963 'unable to run hbck on a secure cluster' to 0.94 > > > Key: HBASE-7814 > URL: https://issues.apache.org/jira/browse/HBASE-7814 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Sergey Shelukhin > Fix For: 0.94.6 > > Attachments: HBASE-7814-v0.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
