Re: Strategies for aggregating data in a HBase table

2011-12-21 Thread Dmitriy Lyubimov
Also re: frontend is always a problem. so far we have a custom data source for this thing in jasper reports, but jdbc eventually is also possible. Looking to see what it takes to mount jpivot to it, but it is more serious endeavor so no big expectations there (unless i pick somebody willing to help

Re: Strategies for aggregating data in a HBase table

2011-12-21 Thread Dmitriy Lyubimov
https://github.com/dlyubimov/HBase-Lattice On Wed, Dec 21, 2011 at 12:13 AM, Dmitriy Lyubimov wrote: > Thomas, > > Sorry for shameless self-promotion. Can you look at our hbase-lattice > project? it is incremental OLAP-ish cube compilation with custom > filtering to optimize for composite key sca

Re: Strategies for aggregating data in a HBase table

2011-12-21 Thread Dmitriy Lyubimov
Thomas, Sorry for shameless self-promotion. Can you look at our hbase-lattice project? it is incremental OLAP-ish cube compilation with custom filtering to optimize for composite key scans. Some rudimental query language as well. Bunch of standard (and not so standard) aggregates for measure data

Re: Strategies for aggregating data in a HBase table

2011-12-01 Thread Jean-Daniel Cryans
Or you could just prefix the row keys. Not sure if this is needed natively, or as a tool on top of HBase. Hive for example could do exactly that for you when Hive partitions are implemented for HBase. J-D On Wed, Nov 30, 2011 at 1:34 PM, Sam Seigal wrote: > What about "partitioning" at a table l

Re: Strategies for aggregating data in a HBase table

2011-11-30 Thread Sam Seigal
What about "partitioning" at a table level. For example, create 12 tables for the given year. Design the row keys however you like, let's say using SHA/MD hashes. Place transactions in the appropriate table and then do aggregations based on that table alone (this is assuming you won't get transacti

Re: Strategies for aggregating data in a HBase table

2011-11-30 Thread Jean-Daniel Cryans
Inline. J-D On Mon, Nov 28, 2011 at 1:55 AM, Steinmaurer Thomas wrote: > Hello, > ... > > While it is an option processing the entire HBase table e.g. every night > when we go live, it probably isn't an option when data volume grows over > the years. So, what options are there for some kind of i

Strategies for aggregating data in a HBase table

2011-11-28 Thread Steinmaurer Thomas
Hello, this has been already discussed a bit in the past, but I'm trying to refresh this thread as this is an important design issue in our HBase evaluation. Basically, the result of our evaluation was that we gonna be happy with what Hadoop/HBase offers for managing our measurement/sensor