[jira] [Created] (HBASE-9012) TestBlockReorder.testBlockLocationReorder fails

2013-07-22 Thread stack (JIRA)
stack created HBASE-9012: Summary: TestBlockReorder.testBlockLocationReorder fails Key: HBASE-9012 URL: https://issues.apache.org/jira/browse/HBASE-9012 Project: HBase Issue Type: Bug

Re: DISCUSS : HFile V3 proposal for tags in 0.96

2013-07-22 Thread Andrew Purtell
[ Reposting from a different account - looks like infra borked LDAP somehow. ] On Fri, Jul 19, 2013 at 4:31 PM, Stack st...@duboce.net wrote: 0.96 is way too late already. I am shooting for end of July as cut-off point when I intend rolling a 0.95.2 RC. We are shooting for a HFileV3 in time

[jira] [Created] (HBASE-9014) TestHLog.testAppendClose fails

2013-07-22 Thread stack (JIRA)
stack created HBASE-9014: Summary: TestHLog.testAppendClose fails Key: HBASE-9014 URL: https://issues.apache.org/jira/browse/HBASE-9014 Project: HBase Issue Type: Bug Reporter: stack

[jira] [Created] (HBASE-9013) Backport HBASE-8994 (Adding log to chaos monkey actions to show what're performed) to 0.94

2013-07-22 Thread Andrew Purtell (JIRA)
Andrew Purtell created HBASE-9013: - Summary: Backport HBASE-8994 (Adding log to chaos monkey actions to show what're performed) to 0.94 Key: HBASE-9013 URL: https://issues.apache.org/jira/browse/HBASE-9013

Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
It seems that current (0.94.6) region assignment on start up policy is not that smart and does not utilize hdfs block locality Are there any open HBase JIRA ticket(s) that addresses the issue? Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail:

[jira] [Created] (HBASE-9015) Client side coprocessor framework

2013-07-22 Thread Andrew Purtell (JIRA)
Andrew Purtell created HBASE-9015: - Summary: Client side coprocessor framework Key: HBASE-9015 URL: https://issues.apache.org/jira/browse/HBASE-9015 Project: HBase Issue Type: New Feature

Re: Smarter Region assignment policy?

2013-07-22 Thread Ted Yu
Please take a look at https://issues.apache.org/jira/browse/HBASE-4755 On Mon, Jul 22, 2013 at 10:56 AM, Vladimir Rodionov vrodio...@carrieriq.com wrote: It seems that current (0.94.6) region assignment on start up policy is not that smart and does not utilize hdfs block locality Are there

RE: Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
Yes, its related but not what I am looking for. Locality index is approaching to 100% after major compaction of all tables. Its often 90% during regular operation of a cluster, but is far below 50% when cluster restarts. Can assignment manager take into account previous region assignments on

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
I faintly remember there was a jira that attempted to assign the regions to the same region servers after a restart based on existing .META. information (if possible). Will try to find that and see why it is not working as expected. -- Lars From: Vladimir

Getting unit tests to pass

2013-07-22 Thread Stack
Below is a state of hbase 0.95/trunk unit tests (Includes a little taxonomy of test failure type definitions). On Andrew's ec2 build box, 0.95 is passing most of the time: http://54.241.6.143/job/HBase-0.95/ http://54.241.6.143/job/HBase-0.95-Hadoop-2/ It is not as good on Apache build box but

[jira] [Created] (HBASE-9016) Cleanup of HRegion (javadoc, unnecessary inits, unnecessary unboxing)

2013-07-22 Thread Alex Newman (JIRA)
Alex Newman created HBASE-9016: -- Summary: Cleanup of HRegion (javadoc, unnecessary inits, unnecessary unboxing) Key: HBASE-9016 URL: https://issues.apache.org/jira/browse/HBASE-9016 Project: HBase

[jira] [Created] (HBASE-9017) Consolidate ChaosMonkey's random objects

2013-07-22 Thread Jimmy Xiang (JIRA)
Jimmy Xiang created HBASE-9017: -- Summary: Consolidate ChaosMonkey's random objects Key: HBASE-9017 URL: https://issues.apache.org/jira/browse/HBASE-9017 Project: HBase Issue Type: Test

[jira] [Created] (HBASE-9018) Add timeouts on all tests in TestHLogSplit

2013-07-22 Thread stack (JIRA)
stack created HBASE-9018: Summary: Add timeouts on all tests in TestHLogSplit Key: HBASE-9018 URL: https://issues.apache.org/jira/browse/HBASE-9018 Project: HBase Issue Type: Bug

[jira] [Resolved] (HBASE-9018) Add timeouts on all tests in TestHLogSplit

2013-07-22 Thread stack (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9018. -- Resolution: Fixed Committed to trunk and to 0.95. Add timeouts on all tests in TestHLogSplit

[jira] [Resolved] (HBASE-8059) Enhance test-patch.sh so that patch can specify hadoop-2.0 as the default profile

2013-07-22 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-8059. --- Resolution: Fixed Enhance test-patch.sh so that patch can specify hadoop-2.0 as the default profile

[jira] [Resolved] (HBASE-7841) Parallelize offline snapshot in DisabledTableSnapshotHandler

2013-07-22 Thread Ted Yu (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-7841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-7841. --- Resolution: Won't Fix Parallelize offline snapshot in DisabledTableSnapshotHandler

[jira] [Created] (HBASE-9019) Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94

2013-07-22 Thread Ted Yu (JIRA)
Ted Yu created HBASE-9019: - Summary: Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94 Key: HBASE-9019 URL: https://issues.apache.org/jira/browse/HBASE-9019

[jira] [Resolved] (HBASE-9019) Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94

2013-07-22 Thread stack (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-9019. -- Resolution: Invalid Resolving issue ill-specified. When I click on the link it takes me to the end of the

RE: Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
OK, you right and I was not. If I do not disable/enables tables after cluster restart all regions seem get assigned correctly. I think its disabling/enabling tables shuffles regions in a bad way. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail:

[jira] [Created] (HBASE-9020) Make hbase-it HBASE_HOME configurable

2013-07-22 Thread Jimmy Xiang (JIRA)
Jimmy Xiang created HBASE-9020: -- Summary: Make hbase-it HBASE_HOME configurable Key: HBASE-9020 URL: https://issues.apache.org/jira/browse/HBASE-9020 Project: HBase Issue Type: Test

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
Hmm... So that's an interesting bug then. Might filing a jira? -- Lars - Original Message - From: Vladimir Rodionov vrodio...@carrieriq.com To: dev@hbase.apache.org dev@hbase.apache.org; lars hofhansl la...@apache.org Cc: Sent: Monday, July 22, 2013 3:30 PM Subject: RE: Smarter

does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
I am doing some tests of append operation (HTable.append(Append append)) and it seems that if cell being appended has multiple versions , all except the last one get deleted. Is it correct behavior or I am doing something wrong in my tests? The test is simple. I have a row with 3 CF and 3

RE: Smarter Region assignment policy?

2013-07-22 Thread Vladimir Rodionov
Is not this https://issues.apache.org/jira/browse/HBASE-6143? Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: lars hofhansl [la...@apache.org] Sent: Monday, July 22, 2013 4:04

Re: does append delete all cell versions except the last one?

2013-07-22 Thread Ted Yu
Can you formulate the code below as a unit test so that other people can try it out ? I pasted the code into a unit test but there were a few unresolved method calls such as createGet(). Cheers On Mon, Jul 22, 2013 at 4:20 PM, Vladimir Rodionov vrodio...@carrieriq.comwrote: Forgot to mention:

[question about replication] how to apply delta from Master to Slave after crash ?

2013-07-22 Thread Demai Ni
hi, folks, I am trying to address a recovery scenario after Master crash. In a simple Master - Slave replication setup for t1, and suddenly Master crashes. there will be a small delta of t1 already changed on Master, but haven't been replicated to Slave yet. Assuming Master's filesystem and

Re: does append delete all cell versions except the last one?

2013-07-22 Thread Ted Yu
Take a look at the release notes for: https://issues.apache.org/jira/browse/HBASE-4583 The fix went to 0.95.0 Cheers On Mon, Jul 22, 2013 at 4:20 PM, Vladimir Rodionov vrodio...@carrieriq.comwrote: Forgot to mention: 0.94.6.1 (CDH 4.3) Best regards, Vladimir Rodionov Principal Platform

Re: Smarter Region assignment policy?

2013-07-22 Thread lars hofhansl
Yep. Seems stale. You can try to comment on it to revive. - Original Message - From: Vladimir Rodionov vrodio...@carrieriq.com To: dev@hbase.apache.org dev@hbase.apache.org; lars hofhansl la...@apache.org Cc: Sent: Monday, July 22, 2013 4:21 PM Subject: RE: Smarter Region assignment

[jira] [Reopened] (HBASE-9019) Port HBASE-8690: Reduce unnecessary getFileStatus hdfs calls in TTL hfile and hlog cleanners to 0.94

2013-07-22 Thread Lars Hofhansl (JIRA)
[ https://issues.apache.org/jira/browse/HBASE-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reopened HBASE-9019: -- Ted fixed the description. Reopening for the backport. Port HBASE-8690: Reduce

[jira] [Created] (HBASE-9021) Create a unit test for ReplicationHLogReaderManager

2013-07-22 Thread Jean-Daniel Cryans (JIRA)
Jean-Daniel Cryans created HBASE-9021: - Summary: Create a unit test for ReplicationHLogReaderManager Key: HBASE-9021 URL: https://issues.apache.org/jira/browse/HBASE-9021 Project: HBase

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Here it is: //--- import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.List; import junit.framework.TestCase; import org.apache.commons.logging.Log; import

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Ted, It is not still clear to me. What I am observing in my tests can be explained only if on each Append we read the latest version of a cell, then put delete marker then put new version. The question still remains: expected behavior or a bug? Best regards, Vladimir Rodionov Principal

[jira] [Created] (HBASE-9022) TestHLogSplit.testIOEOnOutputThread fails

2013-07-22 Thread stack (JIRA)
stack created HBASE-9022: Summary: TestHLogSplit.testIOEOnOutputThread fails Key: HBASE-9022 URL: https://issues.apache.org/jira/browse/HBASE-9022 Project: HBase Issue Type: Bug

Re: does append delete all cell versions except the last one?

2013-07-22 Thread Ted Yu
In 0.94, from HRegion#append(): // Actually write to Memstore now for (Map.EntryStore, ListKeyValue entry : tempMemstore.entrySet()) { Store store = entry.getKey(); size += store.upsert(entry.getValue()); allKVs.addAll(entry.getValue()); }

RE: does append delete all cell versions except the last one?

2013-07-22 Thread Vladimir Rodionov
Thanks, this is what I thought. Best regards, Vladimir Rodionov Principal Platform Engineer Carrier IQ, www.carrieriq.com e-mail: vrodio...@carrieriq.com From: Ted Yu [yuzhih...@gmail.com] Sent: Monday, July 22, 2013 8:07 PM To: dev@hbase.apache.org

[jira] [Created] (HBASE-9023) TestIOFencing.testFencingAroundCompactionAfterWALSync

2013-07-22 Thread stack (JIRA)
stack created HBASE-9023: Summary: TestIOFencing.testFencingAroundCompactionAfterWALSync Key: HBASE-9023 URL: https://issues.apache.org/jira/browse/HBASE-9023 Project: HBase Issue Type: Bug

Re: Getting unit tests to pass

2013-07-22 Thread Stack
By way of illustration of how loaded Apache build boxes can be: Thread LEAK? -, OpenFileDescriptor=174 (was 162) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=4 (was 4), SystemLoadAverage=351 (was 383), ProcessCount=142 (was 144), AvailableMemoryMB=819 (was 892), ConnectionCount=0 (was

Re: Getting unit tests to pass

2013-07-22 Thread Stack
Here is another from tail of https://issues.apache.org/jira/browse/HBASE-5995 2013-07-23 01:23:29,574 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: regionserver.wal.TestLogRolling#testLogRollOnPipelineRestart Thread=39 (was 31) - Thread LEAK? -, OpenFileDescriptor=312 (was 272) -

Re: Getting unit tests to pass

2013-07-22 Thread Stack
nvm. I read the resourcechecker code. It is just printing out before and afters so my speculation that we are up against fd limits is just off. Back to figuring out why tests fail at random St.Ack On Mon, Jul 22, 2013 at 9:50 PM, Stack st...@duboce.net wrote: Here is another from tail

[jira] [Created] (HBASE-9024) TestLogRolling.testLogRollOnPipelineRestart fails/goes zombie

2013-07-22 Thread stack (JIRA)
stack created HBASE-9024: Summary: TestLogRolling.testLogRollOnPipelineRestart fails/goes zombie Key: HBASE-9024 URL: https://issues.apache.org/jira/browse/HBASE-9024 Project: HBase Issue Type: Bug

[jira] [Created] (HBASE-9025) Thread safety issue in MetaScanner

2013-07-22 Thread Lars Hofhansl (JIRA)
Lars Hofhansl created HBASE-9025: Summary: Thread safety issue in MetaScanner Key: HBASE-9025 URL: https://issues.apache.org/jira/browse/HBASE-9025 Project: HBase Issue Type: Bug Affects