Re: Hbase-Hive integration performance issues

Thanks Jean and Sandy. I have hive 0.7.1, and according to this patch https://issues.apache.org/jira/browse/HIVE-1226 at least exact match queries like "...where id = '12345'-123' " or partial pushdown "...where id like "12345%" should work, but I didn't notice it. Matthew. On Mon, Sep 19

Re: prefix compression implementation

So if the HCell or whatever ends up returning ByteBuffers, then that plays straight in to scatter/gather NIO calls, and if some of them are DBB, then so much the merrier. For example, the thrift stuff takes ByteBuffers when its calling for a byte sequence. -ryan On Mon, Sep 19, 2011 at 10:39 PM,

Re: prefix compression implementation

One other thought is that exposing ByteRange, ByteBuffer, and v1 array stuff in Interface seems like you are exposing 'implementation' details that perhaps shouldn't show through. I'm guessing its unavoidable though if the Interface is to be used in a few different contexts: i.e. "v1" has to work

Re: prefix compression implementation

I was just pushing back at the idea of 'turn everything into interfaces! problem solved!', and thinking about what was really necessary to get to where you want to go... On Mon, Sep 19, 2011 at 3:26 PM, Matt Corgan wrote: > Ryan - i answered your question on another thread yesterday.  Will use th

Re: prefix compression implementation

On Mon, Sep 19, 2011 at 3:26 PM, Matt Corgan wrote: > I don't think the name is all that important, though i thought HCell was > less clumsy than KeyValue or KeyValueInterface.  Take a look at this > interface on github: > > https://github.com/hotpads/hbase-prefix-trie/blob/master/src/org/apache/h

Re: HBase as a large, auto-partitioned, sorted, *in-memory* database (was: Re: prefix compression implementation)

Excellent summary Matt. Some notes in the below. On Sun, Sep 18, 2011 at 6:43 PM, Matt Corgan wrote: > ... All of this is relatively easy for the data > and index blocks because they're immutable.  Doing it for the memstore is > another story... > We'd need another data structure completely, wo

Re: Branched 0.92 [WAS -> Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]]

Alright - I'll get the SlabCache into a production usable state before them. I just moved into my awesome new San Diego apartment. Don't have internet yet - tethering off my smartphone. I'll probably be hard to reach via email for a while. On Mon, Sep 19, 2011 at 9:31 AM, Stack wrote: > When we

Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

I applied your patch to 0.92 and trunk. St.Ack On Mon, Sep 19, 2011 at 9:22 PM, Rottinghuis, Joep wrote: > Just filed a jira with patch for this. See HBASE-4447. > > Thanks, > > Joep > > -Original Message- > From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack > Sent:

Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

Yeah, that works too. Generally, I think people go with project.version, but it doesn't really matter ;) -Jesse On Mon, Sep 19, 2011 at 9:22 PM, Rottinghuis, Joep wrote: > Just filed a jira with patch for this. See HBASE-4447. > > Thanks, > > Joep > > -Original Message- > From: saint

RE: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

Just filed a jira with patch for this. See HBASE-4447. Thanks, Joep -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Monday, September 19, 2011 9:21 PM To: dev@hbase.apache.org Subject: Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accu

Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

On Mon, Sep 19, 2011 at 8:53 PM, Jesse Yates wrote: > You should be able to pretty easily set in in the pom (under properties), > and then just use in the tag. > Where are the pom properties Jesse? Thanks, St.Ack > That way whenever you want to to bump version numbers, its one easy change. > >

Re: Running UnitTests before submitting a patch

On Mon, Sep 19, 2011 at 1:31 PM, N Keywal wrote: > Hi, > > Something that would be useful as well is being able to match a given > version of the trunk with the test results. For example: > - Dev pull the last version of the trunk at time t > - Dev do its modification locally to the code > - Dev r

Re: Running UnitTests before submitting a patch

On Mon, Sep 19, 2011 at 1:06 PM, Jesse Yates wrote >> ... will spent (at least on my laptop) about 10.7 seconds setting up the >> cluster, and 7.3 seconds tearing down.  Assuming that we aren't running in >> a separate JVM each test invocation, sharing the same instance of >> HBaseTestingUtility w

Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

You should be able to pretty easily set in in the pom (under properties), and then just use in the tag. That way whenever you want to to bump version numbers, its one easy change. -Jesse Yates On Mon, Sep 19, 2011 at 8:47 PM, Stack wrote: > I changed versions. How would you make hbase.versio

Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

I changed versions. How would you make hbase.version work? Looks like you can't set project.version. I could change it to ${hbase.version} but then how to do the default value? St.Ack On Mon, Sep 19, 2011 at 8:24 PM, Rottinghuis, Joep wrote: > Trunk should probably go to get a newer version as

RE: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

Trunk should probably go to get a newer version as well (0.93?) Can you make the version a property that I can override using -Dhbase.version=0.92-my-own-name? Thanks, Joep -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Monday, Septembe

Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

That makes sense. Let me make the change. St.Ack On Mon, Sep 19, 2011 at 8:02 PM, Rottinghuis, Joep wrote: > Michael, > > Should the version in the pom on the 0.92 branch point to 0.92.0-SNAPSHOT? > If so I can file a bug and supply patch for same. > Or are you updating that only when you get re

Build failed in Jenkins: hbase-0.90 #304

See Changes: [stack] HBASE-4445 Not passing --config when checking if distributed mode or not [stack] HBASE- Miscommit of HBASE-4195 to 0.90 branch (committed as though it were hbase-4423) [stack] HBASE- Miscommit of HBASE-4195 to

RE: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

Michael, Should the version in the pom on the 0.92 branch point to 0.92.0-SNAPSHOT? If so I can file a bug and supply patch for same. Or are you updating that only when you get ready for a release? Thanks, Joep -Original Message- From: Rottinghuis, Joep [mailto:jrottingh...@ebay.com] Se

Re: Running UnitTests before submitting a patch

At Salesforce we annotate tests (with Java annotation). We distinguish between "basic", "normal", and "extended". "basic" can reasonably be expected to be run before every check-in. "normal" is then run by the build system for a small set of changes. "extended" is for very long running tests and

RE: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]

Thanks St.Ack! Joep -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Sunday, September 18, 2011 12:12 PM To: dev@hbase.apache.org Subject: Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Ap

Re: Running UnitTests before submitting a patch

On Mon, Sep 19, 2011 at 1:31 PM, N Keywal wrote: > Hi, > > Something that would be useful as well is being able to match a given > version of the trunk with the test results. For example: > - Dev pull the last version of the trunk at time t > - Dev do its modification locally to the code > - Dev

Re: Running UnitTests before submitting a patch

Also, any test that extends from HBaseClusterTestCase needs to be refactored. This class is already deprecated but it won't be able to take advantage of such a factory because it does the cluster setup differently. E.g.,... public class TestGetRowVersions extends HBaseClusterTestCase { O

Re: prefix compression implementation

Ryan - i answered your question on another thread yesterday. Will use this thread to continue conversation on the KeyValue interface. I don't think the name is all that important, though i thought HCell was less clumsy than KeyValue or KeyValueInterface. Take a look at this interface on github:

Re: SVN commit branches/0.90@1171766

Fixed with HBASE-. I backed out the miscommit. Thanks for catching it Andrew. St.Ack On Fri, Sep 16, 2011 at 6:32 PM, Todd Lipcon wrote: > Agree, looks like accidental commit of nkeywal's change. > > On Fri, Sep 16, 2011 at 6:26 PM, Andrew Purtell wrote: >> https://svn.apache.org/repos/asf

Re: Running UnitTests before submitting a patch

Hi, Something that would be useful as well is being able to match a given version of the trunk with the test results. For example: - Dev pull the last version of the trunk at time t - Dev do its modification locally to the code - Dev runs the unit tests, and compare it to the results on the server

Re: Running UnitTests before submitting a patch

Cool. I'm going to try to get a patch up of the failsafe and package splits up tonight. On Mon, Sep 19, 2011 at 1:18 PM, Doug Meil wrote: > > I'm glad you like the factory idea - it's the only thing I can think of to > keep the same test infrastructure (I.e., not completely re-write > everythin

Re: Running UnitTests before submitting a patch

I'm glad you like the factory idea - it's the only thing I can think of to keep the same test infrastructure (I.e., not completely re-write everything), but still address a sizable performance problem. I'm working on a prototype of it now, I'l attach to a Jira. On 9/19/11 4:06 PM, "Jesse Yate

Re: Running UnitTests before submitting a patch

On Mon, Sep 19, 2011 at 12:18 PM, Doug Meil wrote: > > I'm not too familiar with the Maven Failsafe plugin, but I've been > reviewing the timings of some 'client' unit tests and where the unit test > framework spends it's time... > > Anything that uses HBaseTestingUtility and does something like..

Re: Running UnitTests before submitting a patch

I'm not too familiar with the Maven Failsafe plugin, but I've been reviewing the timings of some 'client' unit tests and where the unit test framework spends it's time... Anything that uses HBaseTestingUtility and does something like... protected void setUp() throws Exception {

Re: Running UnitTests before submitting a patch

Hi, FAIL SAFE PLUGIN: FailSafe Plugin sounds like a great idea. My understanding is that we startup and kill the hdfs, zookeeper and HBase cluster instances for every test suite ( Please correct me if I am wrong here ) . . With FailSafe plugin we would be able to achieve the following things: 1.

Jenkins build is back to normal : HBase-0.92 #2

See

RE: Hbase-Hive integration performance issues

I suffered the same let down a little while ago. I believe this is the relevant JIRA: https://issues.apache.org/jira/browse/HIVE-1643 I'd also like to see Hive be able to limit scans to particular HBase version ranges, but I don't know if that's even planned. Sandy > -Original Message---

Re: Running UnitTests before submitting a patch

On Sun, Sep 18, 2011 at 3:35 PM, Jesse Yates wrote: > I know this was brought up almost a year ago, but it might be time to > revisit breaking out the tests into an integration test suite and a regular > unit test suite. We can use the the maven failsafe plugin ( > http://maven.apache.org/plugins/

Re: Hbase-Hive integration performance issues

(replying to user@, dev@ in BCC) AFAIK the HBase handler doesn't have the wits to understand that you are doing a prefix scan and thus limit the scan to only the required rows. There's a bunch of optimizations like that that need to be done. I'm pretty sure Pig does the same thing, but don't take

Re: Branched 0.92 [WAS -> Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]]

When we kill all the bugs, or end of time, which ever comes first (Would be sweet if it were done for hadoop world or before). St.Ack On Sun, Sep 18, 2011 at 11:36 PM, Li Pi wrote: > When is 0.92 set to be released? > > On Sun, Sep 18, 2011 at 11:34 PM, Stack wrote: >> I just branched 0.92 and s

Re: Branched 0.92 [WAS -> Re: Branching for 0.92 [WAS -> Re: [DISCUSSION] Accumulo, another BigTable clone, has shown up on Apache Incubator as a proposal]]

Here is the svn command to checkout 0.92 branch: svn checkout https://svn.apache.org/repos/asf/hbase/branches/0.92 92hbase Cheers On Sun, Sep 18, 2011 at 11:34 PM, Stack wrote: > I just branched 0.92 and set up a new build for it up on jenkins [1]. > Only bug fixes, doc improvements, and fixes

Re: Issue about the hbase.client.Scan

On Mon, Sep 19, 2011 at 2:03 AM, mac fang wrote: > So Obveriously, the Exception will NOT set since the invokation of setFilter > is later than setBatch. I think it is a bug. > Yes. Mind filing an issue (and perhaps a patch?)? > Second question, i did NOT see any usage of the batch. Instead, i

Build failed in Jenkins: HBase-0.92 #1

See -- [...truncated 1618 lines...] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.316 sec Running org.apache.hadoop.hbase.mapred.TestTableMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipp

Re: question about zookeepers watcher notification

2 might be explained by gc. Could you give a bit more information about case 1? Thanks C - Original Message - From: Ma, Ming To: u...@zookeeper.apache.org Cc: dev@hbase.apache.org Sent: Mon Sep 19 01:28:44 2011 Subject: question about zookeepers watcher notification In the hbase scena

Issue about the hbase.client.Scan

Hi, Guys, When I use the client.Scan class to get a set of rows back from RegionServer, i found one interesting code, the script i used is like: Scan scan = new Scan(Bytes.toBytes("a"), Bytes.toBytes("f")); > scan.addColumn(famA, col1); > scan.setCaching(1); > scan