Re: All builds are recently failing with timeout or fork errors, let's change settings
I see in https://builds.apache.org/computer/ubuntu-2/load-statistics (used for the 0.98 build mentionned by Andrew above) that we have a configuration with 2 executors. It means that jenkins tries to run 2 builds in parallel, each of these builds will trigger its own set of surefire forks. iirc, in the past: - we were not building on these machines, we were using only the hadoop pool of machines - these machines were configured with 1 executor From what I see, there are two sets of machines - H*, for hadoop projects. H0 (for example) is configured with a single executor. - ubuntu*, for everybody: ubuntu2 (for example) is configured with 2 executors. 0.98 and PreCommit-HBASE-Build are configured with: (ubuntu||Hadoop) !jenkins-cloud-4GB !H11 So it depends: lucky = H*. Unlucky = ubuntu* I don't know who changed this, nor why, but may be we should not go to ubuntu* machines. Or, if it's possible, we should have a different config for these machines. On Mon, Jan 19, 2015 at 7:11 PM, Andrew Purtell andrew.purt...@gmail.com wrote: The 0.98 build is still showing this problem (latest as of now at https://builds.apache.org/job/hbase-0.98/803), so I went ahead and made the proposed change, but only to the 0.98 builds. I'll let you know if it provides any improvement. On Sun, Jan 18, 2015 at 10:00 AM, Andrew Purtell andrew.purt...@gmail.com wrote: Forked VMs are being killed in the 0.98 builds. That suggests infrastructure issues. Having only one test execute in a forked runner does mean the finding of a zombie and thread dumps or other state from the runner will identify and characterize a sick test with no unrelated state mixed in. On Jan 17, 2015, at 7:43 PM, Stack st...@duboce.net wrote: Agree, try anything to get our blues back. We add back the //ism after all settles. Do you think something has changed in INFRA Andy? Is it more contended? Or, more likely, is it that we've been committing stuff that has destabilized builds? We had a good streak of blue there for a while. It just took some work fixing breakage and watching jenkins to make sure breakage didn't sneak in, but we've lapsed for sure. St.Ack On Sat, Jan 17, 2015 at 9:19 AM, Dima Spivak dspi...@cloudera.com wrote: Not running tests in parallel will definitely cut down on Surefire flakiness (and in contention that sometimes leads to false failures in resource-hungry tests), but it will probably also balloon test run times to about two hours. Probably worth it in the short term, but we eventually need to do something about some of these heavy tests. -Dima On Friday, January 16, 2015, Andrew Purtell andrew.purt...@gmail.com wrote: You might have missed the larger issue Ted. On Jan 16, 2015, at 4:48 PM, Ted Yu yuzhih...@gmail.com javascript:; wrote: With HBASE-12874, we should get a green build for branch-1.0 FYI On Fri, Jan 16, 2015 at 12:20 PM, Andrew Purtell apurt...@apache.org javascript:; wrote: See BUILDS-49 tracking issues specifically with 0.98 jobs, but I just noticed trunk, branch-1, and branch-1.0 all failed after I checked in a shell doc fix due to a timeout or fork failure. I propose we update all Jenkins jobs to not run tests in parallel, i.e. add -Dsurefire.firstPartForkCount=1 -Dsurefire.secondPartForkCount=1 -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-12922) Post-asciidoc conversion fix-ups part 2
Lars Francke created HBASE-12922: Summary: Post-asciidoc conversion fix-ups part 2 Key: HBASE-12922 URL: https://issues.apache.org/jira/browse/HBASE-12922 Project: HBase Issue Type: Bug Components: documentation Reporter: Lars Francke Assignee: Lars Francke I did read through large parts of the documentation and fixed what I found. Some of it is AsciiDoc stuff, some is contents, some is grammar, some typos fixed etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: No commits to branch-1.0 today please
On Sun, Jan 25, 2015 at 11:00 PM, Enis Söztutar enis@gmail.com wrote: Do you have a script to update CHANGES.txt ? We should bring that in to make_rc.sh. I don't have one. Can only get a report with 100 issues max in it from JIRA which for big releases with 100 items fixed messes w/ our being able to do a complete CHANGES.txt (I did not find a way to page and get items 101+, etc) Usually for me, it is mostly running update CHANGES.txt, run make_rc.sh, then tag. make_rc.sh still of use though only the one hadoop target now? Interesting. St.Ack Enis On Sun, Jan 25, 2015 at 8:36 PM, lars hofhansl la...@apache.org wrote: The way I avoid this for 0.94 now is to create a jenkins job specifically for the RC tag and use that to build/test. That way I only have avoid commits between the time I create the updated CHANGES.txt and when I push the tag (usually 1 or 2 minutes). As soon as the tag is created new commit do not harm, as the tests are off the tag. -- Lars From: Enis Söztutar enis@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org Sent: Sunday, January 25, 2015 5:41 PM Subject: Re: No commits to branch-1.0 today please No worries. Saw that issue but did not have the time to run the tests. Enis On Sun, Jan 25, 2015 at 5:38 PM, Andrew Purtell apurt...@apache.org wrote: Crap, sorry, I did one. It's a new case in TestShell. I didn't see this in time, sorry. No more from hence from me. On Sun, Jan 25, 2015 at 5:21 PM, Enis Söztutar e...@apache.org wrote: I am trying to cut the 1.0.0RC1. Appreciate if no commits come in for today. Enis -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: No commits to branch-1.0 today please
I guess my point was that we do need to test the branch but rather only the tag. So the only important part is to ensure consistency between CHANGES.txt and the tag (and jira, but that can be changed after the fact too).Then I do all testing based on the tag. If I find any issues I start again (i.e. make the necessary changes and update CHANGES.txt and/or move the tag). So as long as there are no issues found during the final testing, which is usually the case at least in 0.94, there is never any issue with other commits. -- Lars From: Andrew Purtell apurt...@apache.org To: dev@hbase.apache.org dev@hbase.apache.org Cc: lars hofhansl la...@apache.org Sent: Monday, January 26, 2015 10:32 AM Subject: Re: No commits to branch-1.0 today please .. and because we are using git, this is all local. I won't be able to commit to the branch if someone has changed upstream, so when that happens I fetch, rebase, redo the tag, then push. Upstream doesn't get messy. On Mon, Jan 26, 2015 at 10:26 AM, Andrew Purtell apurt...@apache.org wrote: I update the version in the POMs, update CHANGES.txt, then tag, then do the rest of the release mechanics, so the time where commits to the branch would be a problem is very short, 5 minutes. On Sun, Jan 25, 2015 at 11:00 PM, Enis Söztutar enis@gmail.com wrote: Do you have a script to update CHANGES.txt ? We should bring that in to make_rc.sh. Usually for me, it is mostly running update CHANGES.txt, run make_rc.sh, then tag. Enis On Sun, Jan 25, 2015 at 8:36 PM, lars hofhansl la...@apache.org wrote: The way I avoid this for 0.94 now is to create a jenkins job specifically for the RC tag and use that to build/test. That way I only have avoid commits between the time I create the updated CHANGES.txt and when I push the tag (usually 1 or 2 minutes). As soon as the tag is created new commit do not harm, as the tests are off the tag. -- Lars From: Enis Söztutar enis@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org Sent: Sunday, January 25, 2015 5:41 PM Subject: Re: No commits to branch-1.0 today please No worries. Saw that issue but did not have the time to run the tests. Enis On Sun, Jan 25, 2015 at 5:38 PM, Andrew Purtell apurt...@apache.org wrote: Crap, sorry, I did one. It's a new case in TestShell. I didn't see this in time, sorry. No more from hence from me. On Sun, Jan 25, 2015 at 5:21 PM, Enis Söztutar e...@apache.org wrote: I am trying to cut the 1.0.0RC1. Appreciate if no commits come in for today. Enis
Re: Async RpcClient
The Async RPC Client has been committed. Thanks to those who helped me with this complex task, especially Stack! :) The existing tests where great to find any hidden issues. If you encounter any issue related to it please report it and mention me in the issue so I can help. (Jurriaan Mous) To quote the Release Note: Retrofit a new, netty-based rpc transport on the client. This client is slightly slower if little contention given the extra tier or so that netty adds and that we block on a Future waiting on the call to finish. This client opens the way for HBase having a native Async API. This client is on by default in master branch (2.0 hbase). It is off in branch-1.0 (hbase-1.1.x). To enable it, set hbase.rpc.client.impl to org.apache.hadoop.hbase.ipc.AsyncRpcClient Lets wait for a while to see if there are any problems found while being the default in master. When proven stable for a while I suggest to remove the old sync-only client on master/trunk first and implement some first additional native async calls in the Table and Admin interface. Then adapt AsyncProcess to native async calls and move scanners over to an optional async call path. Op 18 dec. 2014, om 22:22 heeft Stack st...@duboce.net het volgende geschreven: On Wed, Dec 17, 2014 at 9:44 AM, Jurriaan Mous jurm...@jurmo.us wrote: Hi, I have been working on a Netty 4 based async HBase client to fit better within the event driven server I have been developing. - https://github.com/jurmous/async-hbase-client/tree/HBase-0.99 https://github.com/jurmous/async-hbase-client/tree/HBase-0.99 Recently I have been submitting some patches to make it easier to switch out the RpcClient of HBase. This to enable HBase to use the client itself in all communication. I wanted to do this to use the tests on HBase to check if the client was solid on all edge cases but also to enable HBase to possibly migrate to an async client. These were committed on master and branch-1 https://issues.apache.org/jira/browse/HBASE-12597 https://issues.apache.org/jira/browse/HBASE-12597 https://issues.apache.org/jira/browse/HBASE-12684 https://issues.apache.org/jira/browse/HBASE-12684 Now I am at the next step where I want to contribute back the AsyncRpcClient itself. Sweet. I have opened this issue to add AsyncRpcClient: https://issues.apache.org/jira/browse/HBASE-12684 https://issues.apache.org/jira/browse/HBASE-12684 In the current patch the new async client is the default. 3 questions: Can anyone with a proper Kerberos setup test if the async client works? SASL Digest auth works but I haven’t tested Kerberos yet. Can anyone with know-how on benchmarking test what the performance of this client is compared to the current client? The performance should of course be great in all relevant metrics will it ever be the main client. I can have a go at the above Jurriaan, maybe next week? What will we do with the old RpcClient if the async RpcClient is introduced? It would be great to remove it so hbase can internally base anything async (like AsyncProcess) on the async RPC client and this would not be possible with an also supported sync RPC client. A possible route is to make AsyncRpcClient an option on 1.x and a default on 2.0 branch where we remove the old client. Option in 1.1 and default in 2.0 would be the way to go after we get some signal it is an improvement. When the new AsyncRpcClient will be the default it is possible to introduce callback variants of the Table, Scanner and Admin methods and possibly deprecate batch and other AsyncProcess based calls to replace it with a more flexible batch callback implementation. Excellent. These additions would complement/decorate the new 1.0 Table/Admin? St.Ack
Re: No commits to branch-1.0 today please
.. and because we are using git, this is all local. I won't be able to commit to the branch if someone has changed upstream, so when that happens I fetch, rebase, redo the tag, then push. Upstream doesn't get messy. On Mon, Jan 26, 2015 at 10:26 AM, Andrew Purtell apurt...@apache.org wrote: I update the version in the POMs, update CHANGES.txt, then tag, then do the rest of the release mechanics, so the time where commits to the branch would be a problem is very short, 5 minutes. On Sun, Jan 25, 2015 at 11:00 PM, Enis Söztutar enis@gmail.com wrote: Do you have a script to update CHANGES.txt ? We should bring that in to make_rc.sh. Usually for me, it is mostly running update CHANGES.txt, run make_rc.sh, then tag. Enis On Sun, Jan 25, 2015 at 8:36 PM, lars hofhansl la...@apache.org wrote: The way I avoid this for 0.94 now is to create a jenkins job specifically for the RC tag and use that to build/test. That way I only have avoid commits between the time I create the updated CHANGES.txt and when I push the tag (usually 1 or 2 minutes). As soon as the tag is created new commit do not harm, as the tests are off the tag. -- Lars From: Enis Söztutar enis@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org Sent: Sunday, January 25, 2015 5:41 PM Subject: Re: No commits to branch-1.0 today please No worries. Saw that issue but did not have the time to run the tests. Enis On Sun, Jan 25, 2015 at 5:38 PM, Andrew Purtell apurt...@apache.org wrote: Crap, sorry, I did one. It's a new case in TestShell. I didn't see this in time, sorry. No more from hence from me. On Sun, Jan 25, 2015 at 5:21 PM, Enis Söztutar e...@apache.org wrote: I am trying to cut the 1.0.0RC1. Appreciate if no commits come in for today. Enis
TableMapReduceUtil.initTableMapperJob
Dear all, Issue HBASE-7024 https://issues.apache.org/jira/browse/HBASE-7024 has been closed and the restrictions on types of outputKeyClass and outputValueClass were removed for most overloads of TableMapReduceUtil.initTableMapperJob. These restrictions however remain for the three overloads which take ListScan. I'm not sure if these overloads were added later, or just missed in the previous patch. I can't seem to reopen this issue, perhaps because I don't have permissions as I only just signed up on JIRA. Anyway I attached a patch to the issue, which removes these type restrictions on the three methods taking multiple scans. Best regards, Will Temperley
Re: TableMapReduceUtil.initTableMapperJob
Please open a new JIRA and attach your patch there. Cheers On Mon, Jan 26, 2015 at 6:07 AM, William Temperley willtemper...@gmail.com wrote: Dear all, Issue HBASE-7024 https://issues.apache.org/jira/browse/HBASE-7024 has been closed and the restrictions on types of outputKeyClass and outputValueClass were removed for most overloads of TableMapReduceUtil.initTableMapperJob. These restrictions however remain for the three overloads which take ListScan. I'm not sure if these overloads were added later, or just missed in the previous patch. I can't seem to reopen this issue, perhaps because I don't have permissions as I only just signed up on JIRA. Anyway I attached a patch to the issue, which removes these type restrictions on the three methods taking multiple scans. Best regards, Will Temperley
Re: All builds are recently failing with timeout or fork errors, let's change settings
I removed ubuntu from the label expression for the 0.98 builds. This dropped the available pool of nodes for these jobs from 17 to 10, but the exchange in job stability, if it pans out, will be worth it. On Mon, Jan 26, 2015 at 9:57 AM, Andrew Purtell apurt...@apache.org wrote: I will change the number of executors for the 0.98 builds to 1. Thanks for the tip, N! On Mon, Jan 26, 2015 at 8:45 AM, Nicolas Liochon nkey...@gmail.com wrote: I see in https://builds.apache.org/computer/ubuntu-2/load-statistics (used for the 0.98 build mentionned by Andrew above) that we have a configuration with 2 executors. It means that jenkins tries to run 2 builds in parallel, each of these builds will trigger its own set of surefire forks. iirc, in the past: - we were not building on these machines, we were using only the hadoop pool of machines - these machines were configured with 1 executor From what I see, there are two sets of machines - H*, for hadoop projects. H0 (for example) is configured with a single executor. - ubuntu*, for everybody: ubuntu2 (for example) is configured with 2 executors. 0.98 and PreCommit-HBASE-Build are configured with: (ubuntu||Hadoop) !jenkins-cloud-4GB !H11 So it depends: lucky = H*. Unlucky = ubuntu* I don't know who changed this, nor why, but may be we should not go to ubuntu* machines. Or, if it's possible, we should have a different config for these machines. On Mon, Jan 19, 2015 at 7:11 PM, Andrew Purtell andrew.purt...@gmail.com wrote: The 0.98 build is still showing this problem (latest as of now at https://builds.apache.org/job/hbase-0.98/803), so I went ahead and made the proposed change, but only to the 0.98 builds. I'll let you know if it provides any improvement. On Sun, Jan 18, 2015 at 10:00 AM, Andrew Purtell andrew.purt...@gmail.com wrote: Forked VMs are being killed in the 0.98 builds. That suggests infrastructure issues. Having only one test execute in a forked runner does mean the finding of a zombie and thread dumps or other state from the runner will identify and characterize a sick test with no unrelated state mixed in. On Jan 17, 2015, at 7:43 PM, Stack st...@duboce.net wrote: Agree, try anything to get our blues back. We add back the //ism after all settles. Do you think something has changed in INFRA Andy? Is it more contended? Or, more likely, is it that we've been committing stuff that has destabilized builds? We had a good streak of blue there for a while. It just took some work fixing breakage and watching jenkins to make sure breakage didn't sneak in, but we've lapsed for sure. St.Ack On Sat, Jan 17, 2015 at 9:19 AM, Dima Spivak dspi...@cloudera.com wrote: Not running tests in parallel will definitely cut down on Surefire flakiness (and in contention that sometimes leads to false failures in resource-hungry tests), but it will probably also balloon test run times to about two hours. Probably worth it in the short term, but we eventually need to do something about some of these heavy tests. -Dima On Friday, January 16, 2015, Andrew Purtell andrew.purt...@gmail.com wrote: You might have missed the larger issue Ted. On Jan 16, 2015, at 4:48 PM, Ted Yu yuzhih...@gmail.com javascript:; wrote: With HBASE-12874, we should get a green build for branch-1.0 FYI On Fri, Jan 16, 2015 at 12:20 PM, Andrew Purtell apurt...@apache.org javascript:; wrote: See BUILDS-49 tracking issues specifically with 0.98 jobs, but I just noticed trunk, branch-1, and branch-1.0 all failed after I checked in a shell doc fix due to a timeout or fork failure. I propose we update all Jenkins jobs to not run tests in parallel, i.e. add -Dsurefire.firstPartForkCount=1 -Dsurefire.secondPartForkCount=1 -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: No commits to branch-1.0 today please
I update the version in the POMs, update CHANGES.txt, then tag, then do the rest of the release mechanics, so the time where commits to the branch would be a problem is very short, 5 minutes. On Sun, Jan 25, 2015 at 11:00 PM, Enis Söztutar enis@gmail.com wrote: Do you have a script to update CHANGES.txt ? We should bring that in to make_rc.sh. Usually for me, it is mostly running update CHANGES.txt, run make_rc.sh, then tag. Enis On Sun, Jan 25, 2015 at 8:36 PM, lars hofhansl la...@apache.org wrote: The way I avoid this for 0.94 now is to create a jenkins job specifically for the RC tag and use that to build/test. That way I only have avoid commits between the time I create the updated CHANGES.txt and when I push the tag (usually 1 or 2 minutes). As soon as the tag is created new commit do not harm, as the tests are off the tag. -- Lars From: Enis Söztutar enis@gmail.com To: dev@hbase.apache.org dev@hbase.apache.org Sent: Sunday, January 25, 2015 5:41 PM Subject: Re: No commits to branch-1.0 today please No worries. Saw that issue but did not have the time to run the tests. Enis On Sun, Jan 25, 2015 at 5:38 PM, Andrew Purtell apurt...@apache.org wrote: Crap, sorry, I did one. It's a new case in TestShell. I didn't see this in time, sorry. No more from hence from me. On Sun, Jan 25, 2015 at 5:21 PM, Enis Söztutar e...@apache.org wrote: I am trying to cut the 1.0.0RC1. Appreciate if no commits come in for today. Enis -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: All builds are recently failing with timeout or fork errors, let's change settings
I will change the number of executors for the 0.98 builds to 1. Thanks for the tip, N! On Mon, Jan 26, 2015 at 8:45 AM, Nicolas Liochon nkey...@gmail.com wrote: I see in https://builds.apache.org/computer/ubuntu-2/load-statistics (used for the 0.98 build mentionned by Andrew above) that we have a configuration with 2 executors. It means that jenkins tries to run 2 builds in parallel, each of these builds will trigger its own set of surefire forks. iirc, in the past: - we were not building on these machines, we were using only the hadoop pool of machines - these machines were configured with 1 executor From what I see, there are two sets of machines - H*, for hadoop projects. H0 (for example) is configured with a single executor. - ubuntu*, for everybody: ubuntu2 (for example) is configured with 2 executors. 0.98 and PreCommit-HBASE-Build are configured with: (ubuntu||Hadoop) !jenkins-cloud-4GB !H11 So it depends: lucky = H*. Unlucky = ubuntu* I don't know who changed this, nor why, but may be we should not go to ubuntu* machines. Or, if it's possible, we should have a different config for these machines. On Mon, Jan 19, 2015 at 7:11 PM, Andrew Purtell andrew.purt...@gmail.com wrote: The 0.98 build is still showing this problem (latest as of now at https://builds.apache.org/job/hbase-0.98/803), so I went ahead and made the proposed change, but only to the 0.98 builds. I'll let you know if it provides any improvement. On Sun, Jan 18, 2015 at 10:00 AM, Andrew Purtell andrew.purt...@gmail.com wrote: Forked VMs are being killed in the 0.98 builds. That suggests infrastructure issues. Having only one test execute in a forked runner does mean the finding of a zombie and thread dumps or other state from the runner will identify and characterize a sick test with no unrelated state mixed in. On Jan 17, 2015, at 7:43 PM, Stack st...@duboce.net wrote: Agree, try anything to get our blues back. We add back the //ism after all settles. Do you think something has changed in INFRA Andy? Is it more contended? Or, more likely, is it that we've been committing stuff that has destabilized builds? We had a good streak of blue there for a while. It just took some work fixing breakage and watching jenkins to make sure breakage didn't sneak in, but we've lapsed for sure. St.Ack On Sat, Jan 17, 2015 at 9:19 AM, Dima Spivak dspi...@cloudera.com wrote: Not running tests in parallel will definitely cut down on Surefire flakiness (and in contention that sometimes leads to false failures in resource-hungry tests), but it will probably also balloon test run times to about two hours. Probably worth it in the short term, but we eventually need to do something about some of these heavy tests. -Dima On Friday, January 16, 2015, Andrew Purtell andrew.purt...@gmail.com wrote: You might have missed the larger issue Ted. On Jan 16, 2015, at 4:48 PM, Ted Yu yuzhih...@gmail.com javascript:; wrote: With HBASE-12874, we should get a green build for branch-1.0 FYI On Fri, Jan 16, 2015 at 12:20 PM, Andrew Purtell apurt...@apache.org javascript:; wrote: See BUILDS-49 tracking issues specifically with 0.98 jobs, but I just noticed trunk, branch-1, and branch-1.0 all failed after I checked in a shell doc fix due to a timeout or fork failure. I propose we update all Jenkins jobs to not run tests in parallel, i.e. add -Dsurefire.firstPartForkCount=1 -Dsurefire.secondPartForkCount=1 -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-12923) ResultScanner is not closed in ModifyTableHandler#removeReplicaColumnsIfNeeded()
Ted Yu created HBASE-12923:
--
Summary: ResultScanner is not closed in
ModifyTableHandler#removeReplicaColumnsIfNeeded()
Key: HBASE-12923
URL: https://issues.apache.org/jira/browse/HBASE-12923
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
In ModifyTableHandler#removeReplicaColumnsIfNeeded():
{code}
ResultScanner resScanner = metaTable.getScanner(scan);
for (Result result : resScanner) {
{code}
The ResultScanner is not closed upon exit from the method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-11431) Add support of running from command line for 'hbase shell'
[ https://issues.apache.org/jira/browse/HBASE-11431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11431. Resolution: Not a Problem Add support of running from command line for 'hbase shell' -- Key: HBASE-11431 URL: https://issues.apache.org/jira/browse/HBASE-11431 Project: HBase Issue Type: New Feature Components: Admin Affects Versions: 0.89-fb Reporter: @deprecated Yi Deng Priority: Minor Labels: shell Fix For: 0.89-fb Add support of running from command line for 'hbase shell'. Now you can execute shell command from the bash like this: bin/hbase shell --exec='scan .META' The result can be piped to grep or other command. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12925) Use acl cache for doing access control checks in prepare and clean phases of Bulkloading.
Srikanth Srungarapu created HBASE-12925:
---
Summary: Use acl cache for doing access control checks in prepare
and clean phases of Bulkloading.
Key: HBASE-12925
URL: https://issues.apache.org/jira/browse/HBASE-12925
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Currently, prepareBulkLoad and cleanupBulkLoad are using hasSomeAccess, which
performs scan on ACL table, instead of using TableAuthManager. Also, the method
hasSomeAccess has a logical error, as it doesn't filter the acl scan results
by the current active user. More specifically
{code}
for (UserPermission userPerm: perms) {
for (Action userAction: userPerm.getActions()) {
if (userAction.equals(action)) {
return AuthResult.allow(method, Access allowed, requestUser,
action, tableName, null, null);
}
}
}
{code}
The if clause ideally should be having something like
userPerm.getUser.equals(requestUser). This issue will help us in getting rid of
this problematic implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[VOTE] The 1st HBase 0.98.10 release candidate (RC0) is available, vote closing 2/2/2015
The 1st HBase 0.98.10 release candidate (RC0) is available for download at http://people.apache.org/~apurtell/0.98.10RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1057/ Signed with my code signing key D5365CCD. The issues resolved in this release can be found at http://s.apache.org/7hO Please try out the candidate and vote +1/-1 by midnight Pacific Time (00:00 -0800 GMT) on February 2 on whether or not we should release this as 0.98.10. Three +1 votes from PMC will be required to release. -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
[jira] [Created] (HBASE-12928) BaseLoadBalancer#needsBalance only checks the sloppiness of region count before balancing
cuijianwei created HBASE-12928: -- Summary: BaseLoadBalancer#needsBalance only checks the sloppiness of region count before balancing Key: HBASE-12928 URL: https://issues.apache.org/jira/browse/HBASE-12928 Project: HBase Issue Type: Improvement Components: Balancer Affects Versions: 0.99.2 Reporter: cuijianwei Priority: Minor BaseLoadBalancer#needsBalance will be invoked to judge whether needs to do balancing. StochasticLoadBalancer do balancing by considering region count skew cost, read/write request cost, locality cost, etc. However, it seems that only sloppiness of region count is checked in BaseLoadBalancer#needsBalance, there may be cases that request/locality cost is high when region count is even, this will skip the actual balancing so that can't achieve lower cost. There, Do we need to check sloppiness of other factors(read/write request, locality, etc) in needsBalance? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12926) Backport HBASE-12688 (Update site with a bootstrap-based UI) for HBASE-12918
Andrew Purtell created HBASE-12926: -- Summary: Backport HBASE-12688 (Update site with a bootstrap-based UI) for HBASE-12918 Key: HBASE-12926 URL: https://issues.apache.org/jira/browse/HBASE-12926 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Backport HBASE-12688 (Update site with a bootstrap-based UI) for HBASE-12918 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12927) TestFromClientSide#testScanMetrics() failing due to duplicate createTable commands
Jonathan Lawlor created HBASE-12927: --- Summary: TestFromClientSide#testScanMetrics() failing due to duplicate createTable commands Key: HBASE-12927 URL: https://issues.apache.org/jira/browse/HBASE-12927 Project: HBase Issue Type: Bug Reporter: Jonathan Lawlor Assignee: Jonathan Lawlor Priority: Minor Fix For: 1.1.0 The testScanMetrics test is failing because a createTable command was not removed in the branch-1 patch from HBASE-7541. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12669) Have compaction scanner save info about delete markers
[ https://issues.apache.org/jira/browse/HBASE-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du reopened HBASE-12669: -- Have compaction scanner save info about delete markers -- Key: HBASE-12669 URL: https://issues.apache.org/jira/browse/HBASE-12669 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jonathan Hsieh Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-12669-V2.diff, HBASE-12669-V3.diff, HBASE-12669-V4.diff, HBASE-12669-V5.diff, HBASE-12669-V6.diff, HBASE-12669.diff for the native mob compaction, we will need a scanner pass used the major compaction that records the delete markers in the mob-enabled columns. This would implement that core section. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12669) Have compaction scanner save info about delete markers
[ https://issues.apache.org/jira/browse/HBASE-12669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du resolved HBASE-12669. -- Resolution: Fixed Have compaction scanner save info about delete markers -- Key: HBASE-12669 URL: https://issues.apache.org/jira/browse/HBASE-12669 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jonathan Hsieh Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-12669-V2.diff, HBASE-12669-V3.diff, HBASE-12669-V4.diff, HBASE-12669-V5.diff, HBASE-12669-V6.diff, HBASE-12669.diff for the native mob compaction, we will need a scanner pass used the major compaction that records the delete markers in the mob-enabled columns. This would implement that core section. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12921) Port HBASE-5356 'region_mover.rb can hang if table region it belongs to is deleted' to 0.94
Liu Shaohui created HBASE-12921: --- Summary: Port HBASE-5356 'region_mover.rb can hang if table region it belongs to is deleted' to 0.94 Key: HBASE-12921 URL: https://issues.apache.org/jira/browse/HBASE-12921 Project: HBase Issue Type: Bug Affects Versions: 0.94.11 Environment: Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.94.28 This is backport of HBASE-5356 'region_mover.rb can hang if table region it belongs to is deleted' to 0.94. [~lhofhansl] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Second release candidate for HBase 1.0.0 (RC1) is available. Please vote by Feb 08 2015
Sigh. Consider this RC in the bottom of the ocean. See https://issues.apache.org/jira/browse/HBASE-12919 Enis On Mon, Jan 26, 2015 at 12:11 AM, Enis Söztutar e...@apache.org wrote: I gives me great pleasure to announce that the second release candidate for the release 1.0.0 (HBase-1.0.0RC1), is available for download at https://dist.apache.org/repos/dist/dev/hbase/hbase-1.0.0RC1/ Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1056 Signed with my code signing key E964B5FF. Can be found here: https://people.apache.org/keys/committer/enis.asc Signed tag in the repository can be found here: https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=46edac369542227e23c6e0b5edfe3aa68ca264a3 HBase 1.0.0 is the next stable release, and the start of semantic versioned releases (See [1]). The theme of 1.0.0 release is to become a stable base for future 1.x series of releases. We aim to achieve at least the same level of stability of 0.98 releases. 1.0.0 release contains 155 fixes on top of 0.99.2 release. Together with the previous 0.99.x releases, major changes in 1.0.0 are listed (but not limited to) below. Note that all 0.99.x releases are developer preview releases, and will NOT be supported in any form. API Cleanup and changes 1.0.0 introduces new APIs, and deprecates some of commonly-used client side APIs (HTableInterface, HTable and HBaseAdmin). We advise to update your application to use the new style of APIs, since deprecated APIs might be removed in future releases (2.x). See [2] and [3] for an overview of changes. All Client side API's are marked with InterfaceAudience.Public class, indicating that the class/method is an official client API for HBase. All 1.x releases are planned to be API compatible for these classes. See [1] for an overview. Master runs a Region Server as well Starting with 1.0.0, the HBase master server and backup master servers will also act as a region server. RPC port and info port for web UI is shared for the master and region server roles. Active master can host regions of defined tables if configured (disabled by default). Backup masters will not host regions. Read availability using timeline consistent region replicas This release contains Phase 1 items for experimental Read availability using timeline consistent region replicas feature. A region can be hosted in multiple region servers in read-only mode. One of the replicas for the region will be primary, accepting writes, and other replicas will be sharing the same data files. Read requests can be done against any replica for the region with backup RPCs for high availability with timeline consistency guarantees. More information can be found at HBASE-10070. Online config change and other forward ports from 0.89-fb branch HBASE-12147 forward ported online config change which enables some of the configuration from the server to be reloaded without restarting the region servers. Other notable improvements in 1.0.0 (including previous 0.99.x) are - A new web skin in time for 1.0 (http://hbase.apache.org) - Automatic tuning of global memstore and block cache sizes - Various security, tags and visibility labels improvements - Bucket cache improvements (usability and compressed data blocks) - A new pluggable replication endpoint to plug in to HBase's inter-cluster replication to replicate to a custom data store - A Dockerfile to easily build and run HBase from source - Truncate table command - Region assignment to use hbase:meta table instead of zookeeper for faster region assignment (disabled by default) - Extensive documentation improvements - [HBASE-12511] - namespace permissions - add support from table creation privilege in a namespace 'C' - [HBASE-12568] - Adopt Semantic Versioning and document it in the book - [HBASE-12640] - Add Thrift-over-HTTPS and doAs support for Thrift Server - [HBASE-12651] - Backport HBASE-12559 'Provide LoadBalancer with online configuration capability' to branch-1 - [HBASE-10560] - Per cell TTLs - [HBASE-11997] - CopyTable with bulkload - [HBASE-11990] - Make setting the start and stop row for a specific prefix easier - [HBASE-12220] - Add hedgedReads and hedgedReadWins metrics - [HBASE-12090] - Bytes: more Unsafe, more Faster - [HBASE-12032] - Script to stop regionservers via RPC - [HBASE-11907] - Use the joni byte[] regex engine in place of j.u.regex in RegexStringComparator - [HBASE-11796] - Add client support for atomic checkAndMutate - [HBASE-11804] - Raise default heap size if unspecified - [HBASE-11890] - HBase REST Client is hard coded to http protocol - [HBASE-12126] - Region server coprocessor endpoint - [HBASE-12183] - FuzzyRowFilter doesn't support reverse scans - [HBASE-12075] - Preemptive Fast Fail - [HBASE-12354] -
Re: Second release candidate for HBase 1.0.0 (RC1) is available. Please vote by Feb 08 2015
I'll spin a new one tomorrow. Enis On Mon, Jan 26, 2015 at 12:26 AM, Enis Söztutar e...@apache.org wrote: Sigh. Consider this RC in the bottom of the ocean. See https://issues.apache.org/jira/browse/HBASE-12919 Enis On Mon, Jan 26, 2015 at 12:11 AM, Enis Söztutar e...@apache.org wrote: I gives me great pleasure to announce that the second release candidate for the release 1.0.0 (HBase-1.0.0RC1), is available for download at https://dist.apache.org/repos/dist/dev/hbase/hbase-1.0.0RC1/ Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1056 Signed with my code signing key E964B5FF. Can be found here: https://people.apache.org/keys/committer/enis.asc Signed tag in the repository can be found here: https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=46edac369542227e23c6e0b5edfe3aa68ca264a3 HBase 1.0.0 is the next stable release, and the start of semantic versioned releases (See [1]). The theme of 1.0.0 release is to become a stable base for future 1.x series of releases. We aim to achieve at least the same level of stability of 0.98 releases. 1.0.0 release contains 155 fixes on top of 0.99.2 release. Together with the previous 0.99.x releases, major changes in 1.0.0 are listed (but not limited to) below. Note that all 0.99.x releases are developer preview releases, and will NOT be supported in any form. API Cleanup and changes 1.0.0 introduces new APIs, and deprecates some of commonly-used client side APIs (HTableInterface, HTable and HBaseAdmin). We advise to update your application to use the new style of APIs, since deprecated APIs might be removed in future releases (2.x). See [2] and [3] for an overview of changes. All Client side API's are marked with InterfaceAudience.Public class, indicating that the class/method is an official client API for HBase. All 1.x releases are planned to be API compatible for these classes. See [1] for an overview. Master runs a Region Server as well Starting with 1.0.0, the HBase master server and backup master servers will also act as a region server. RPC port and info port for web UI is shared for the master and region server roles. Active master can host regions of defined tables if configured (disabled by default). Backup masters will not host regions. Read availability using timeline consistent region replicas This release contains Phase 1 items for experimental Read availability using timeline consistent region replicas feature. A region can be hosted in multiple region servers in read-only mode. One of the replicas for the region will be primary, accepting writes, and other replicas will be sharing the same data files. Read requests can be done against any replica for the region with backup RPCs for high availability with timeline consistency guarantees. More information can be found at HBASE-10070. Online config change and other forward ports from 0.89-fb branch HBASE-12147 forward ported online config change which enables some of the configuration from the server to be reloaded without restarting the region servers. Other notable improvements in 1.0.0 (including previous 0.99.x) are - A new web skin in time for 1.0 (http://hbase.apache.org) - Automatic tuning of global memstore and block cache sizes - Various security, tags and visibility labels improvements - Bucket cache improvements (usability and compressed data blocks) - A new pluggable replication endpoint to plug in to HBase's inter-cluster replication to replicate to a custom data store - A Dockerfile to easily build and run HBase from source - Truncate table command - Region assignment to use hbase:meta table instead of zookeeper for faster region assignment (disabled by default) - Extensive documentation improvements - [HBASE-12511] - namespace permissions - add support from table creation privilege in a namespace 'C' - [HBASE-12568] - Adopt Semantic Versioning and document it in the book - [HBASE-12640] - Add Thrift-over-HTTPS and doAs support for Thrift Server - [HBASE-12651] - Backport HBASE-12559 'Provide LoadBalancer with online configuration capability' to branch-1 - [HBASE-10560] - Per cell TTLs - [HBASE-11997] - CopyTable with bulkload - [HBASE-11990] - Make setting the start and stop row for a specific prefix easier - [HBASE-12220] - Add hedgedReads and hedgedReadWins metrics - [HBASE-12090] - Bytes: more Unsafe, more Faster - [HBASE-12032] - Script to stop regionservers via RPC - [HBASE-11907] - Use the joni byte[] regex engine in place of j.u.regex in RegexStringComparator - [HBASE-11796] - Add client support for atomic checkAndMutate - [HBASE-11804] - Raise default heap size if unspecified - [HBASE-11890] - HBase REST Client is hard coded to http protocol - [HBASE-12126] - Region server coprocessor endpoint -
Second release candidate for HBase 1.0.0 (RC1) is available. Please vote by Feb 08 2015
I gives me great pleasure to announce that the second release candidate for the release 1.0.0 (HBase-1.0.0RC1), is available for download at https://dist.apache.org/repos/dist/dev/hbase/hbase-1.0.0RC1/ Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1056 Signed with my code signing key E964B5FF. Can be found here: https://people.apache.org/keys/committer/enis.asc Signed tag in the repository can be found here: https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=tag;h=46edac369542227e23c6e0b5edfe3aa68ca264a3 HBase 1.0.0 is the next stable release, and the start of semantic versioned releases (See [1]). The theme of 1.0.0 release is to become a stable base for future 1.x series of releases. We aim to achieve at least the same level of stability of 0.98 releases. 1.0.0 release contains 155 fixes on top of 0.99.2 release. Together with the previous 0.99.x releases, major changes in 1.0.0 are listed (but not limited to) below. Note that all 0.99.x releases are developer preview releases, and will NOT be supported in any form. API Cleanup and changes 1.0.0 introduces new APIs, and deprecates some of commonly-used client side APIs (HTableInterface, HTable and HBaseAdmin). We advise to update your application to use the new style of APIs, since deprecated APIs might be removed in future releases (2.x). See [2] and [3] for an overview of changes. All Client side API's are marked with InterfaceAudience.Public class, indicating that the class/method is an official client API for HBase. All 1.x releases are planned to be API compatible for these classes. See [1] for an overview. Master runs a Region Server as well Starting with 1.0.0, the HBase master server and backup master servers will also act as a region server. RPC port and info port for web UI is shared for the master and region server roles. Active master can host regions of defined tables if configured (disabled by default). Backup masters will not host regions. Read availability using timeline consistent region replicas This release contains Phase 1 items for experimental Read availability using timeline consistent region replicas feature. A region can be hosted in multiple region servers in read-only mode. One of the replicas for the region will be primary, accepting writes, and other replicas will be sharing the same data files. Read requests can be done against any replica for the region with backup RPCs for high availability with timeline consistency guarantees. More information can be found at HBASE-10070. Online config change and other forward ports from 0.89-fb branch HBASE-12147 forward ported online config change which enables some of the configuration from the server to be reloaded without restarting the region servers. Other notable improvements in 1.0.0 (including previous 0.99.x) are - A new web skin in time for 1.0 (http://hbase.apache.org) - Automatic tuning of global memstore and block cache sizes - Various security, tags and visibility labels improvements - Bucket cache improvements (usability and compressed data blocks) - A new pluggable replication endpoint to plug in to HBase's inter-cluster replication to replicate to a custom data store - A Dockerfile to easily build and run HBase from source - Truncate table command - Region assignment to use hbase:meta table instead of zookeeper for faster region assignment (disabled by default) - Extensive documentation improvements - [HBASE-12511] - namespace permissions - add support from table creation privilege in a namespace 'C' - [HBASE-12568] - Adopt Semantic Versioning and document it in the book - [HBASE-12640] - Add Thrift-over-HTTPS and doAs support for Thrift Server - [HBASE-12651] - Backport HBASE-12559 'Provide LoadBalancer with online configuration capability' to branch-1 - [HBASE-10560] - Per cell TTLs - [HBASE-11997] - CopyTable with bulkload - [HBASE-11990] - Make setting the start and stop row for a specific prefix easier - [HBASE-12220] - Add hedgedReads and hedgedReadWins metrics - [HBASE-12090] - Bytes: more Unsafe, more Faster - [HBASE-12032] - Script to stop regionservers via RPC - [HBASE-11907] - Use the joni byte[] regex engine in place of j.u.regex in RegexStringComparator - [HBASE-11796] - Add client support for atomic checkAndMutate - [HBASE-11804] - Raise default heap size if unspecified - [HBASE-11890] - HBase REST Client is hard coded to http protocol - [HBASE-12126] - Region server coprocessor endpoint - [HBASE-12183] - FuzzyRowFilter doesn't support reverse scans - [HBASE-12075] - Preemptive Fast Fail - [HBASE-12354] - Update dependencies in time for 1.0 release - [HBASE-12363] - Improve how KEEP_DELETED_CELLS works with MIN_VERSIONS - [HBASE-12434] - Add a command to compact all the regions in a regionserver - [HBASE-8707] - Add LongComparator for filter - [HBASE-12286] - [shell] Add
[jira] [Created] (HBASE-12920) hadoopqa should compile with different hadoop versions
Enis Soztutar created HBASE-12920:
-
Summary: hadoopqa should compile with different hadoop versions
Key: HBASE-12920
URL: https://issues.apache.org/jira/browse/HBASE-12920
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Fix For: 2.0.0
From time to time, we break compilation with hadoop-2.4 or other earlier
versions, and only realize that at the time of a release candidate.
We should fix hadoopqa to do the compilation for us.
What I have locally is something like this:
{code}
HADOOP2_VERSIONS=2.2.0 2.3.0 2.4.0 2.5.0 2.6.0
function buildWithHadoop2 {
for HADOOP2_VERSION in $HADOOP2_VERSIONS ; do
echo
echo # BUILDING $ARTIFACT WITH HADOOP 2 VERSION $HADOOP2_VERSION #
echo
echo mvn clean install -DskipTests -Dhadoop-two.version=$HADOOP2_VERSION
mvn clean install -DskipTests -Dhadoop-two.version=$HADOOP2_VERSION
done
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12919) Compilation with Hadoop-2.4- is broken again
Enis Soztutar created HBASE-12919:
-
Summary: Compilation with Hadoop-2.4- is broken again
Key: HBASE-12919
URL: https://issues.apache.org/jira/browse/HBASE-12919
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
This is from the 1.0.0RC1 testing. I should have done the compilation before
the RC. Sorry for the noise.
{code}
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/enis/projects/rc-test/hbase-1.0.0/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java:[98,18]
error: method authorize in class ProxyUsers cannot be applied to given types;
{code}
HBASE-12640 seems to be the offender. [~srikanth235], [~jxiang] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12924) HRegionServer#MovedRegionsCleaner Chore does not start
Jonathan Lawlor created HBASE-12924:
---
Summary: HRegionServer#MovedRegionsCleaner Chore does not start
Key: HBASE-12924
URL: https://issues.apache.org/jira/browse/HBASE-12924
Project: HBase
Issue Type: Bug
Reporter: Jonathan Lawlor
Priority: Minor
This issue is very similar to the one described in HBASE-11354. The method
createAndStart in MovedRegionsCleaner creates an instance of the chore but
never starts the underlying thread:
{code}
static MovedRegionsCleaner createAndStart(HRegionServer rs){
Stoppable stoppable = new Stoppable() {
private volatile boolean isStopped = false;
@Override public void stop(String why) { isStopped = true;}
@Override public boolean isStopped() {return isStopped;}
};
return new MovedRegionsCleaner(rs, stoppable);
}
{code}
Nobody has noticed this issue so far so the Chore may not be that important
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
