Hallo,
It 's weird that hbase aggregate functions don't use MapReduce, this means
that the performance will be very poor.
Is it a must to use coprocessors?
Is there a much easier way to improve the functions' performance ?
Why would performance be poor? I am not dealing a long time with these
coprocessors and am still testing
a lot, but in my perception its much more lightweight on the other hand.
Actually, depending on your row key design and request range, the load
is distributed across the region servers.
Each will handle aggregation for its own key range. The client, invoking
the coprpocessor then must merge the results, which
can be seen as sort of reduce function.
I think one has to precisely think about its own requirements. I am not
sure how and even if M/R Jobs can work for real-time scenarios.
Here, coprocessors seem to be a good alternative, which could also be
limited, depending of how many rows you need to iterate in the
coprocessor for the kind of data you have and expect to request.
However, having the data processed in a M/R job before and persistent
would be faster for the single client request because you only need to
fetch the aggregated data that then already exists. But how recent is
data at this time? Does it change frequently and aggregation results
must be as recent as the data it reflects? Could be a con against M/R...
Regards
tom
Am 25.11.2012 07:26, schrieb Wei Tan:
Actually coprocessor can be used to implement MR-like function, while not
using Hadoop framework.
Best Regards,
Wei
Wei Tan
Research Staff Member
IBM T. J. Watson Research Center
Yorktown Heights, NY 10598
w...@us.ibm.com; 914-784-6752
From: Dalia Sobhy <dalia.mohso...@hotmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
Date: 11/24/2012 01:33 PM
Subject: RE: Hbase MapReduce
It 's weird that hbase aggregate functions don't use MapReduce, this means
that the performance will be very poor.
Is it a must to use coprocessors?
Is there a much easier way to improve the functions' performance ?
CC: user@hbase.apache.org
From: michael_se...@hotmail.com
Subject: Re: Hbase MapReduce
Date: Sat, 24 Nov 2012 12:05:45 -0600
To: user@hbase.apache.org
Do you think it would be a good idea to temper the use of CoProcessors?
This kind of reminds me of when people first started using stored
procedures...
Sent from a remote device. Please excuse any typos...
Mike Segel
On Nov 24, 2012, at 11:46 AM, tom <t...@arcor.de> wrote:
Hi, but you do not need to us M/R. You could also use coprocessors.
See this site:
https://blogs.apache.org/hbase/entry/coprocessor_introduction
-> in the section "Endpoints"
An aggregation coprocessor ships with hbase that should match your
requirements.
You just need to load it and eventually you can access it from HTable:
HTable.coprocessorExec(..) <
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#coprocessorExec%28java.lang.Class,%20byte
[],%20byte[],%20org.apache.hadoop.hbase.client.coprocessor.Batch.Call,%20org.apache.hadoop.hbase.client.coprocessor.Batch.Callback%29>
Regards
tom
Am 24.11.2012 18:32, schrieb Marcos Ortiz:
Regards, Dalia.
You have to use MapReduce for that.
In the HBase in Practice´s book, there are lot of great examples for
this.
On 11/24/2012 12:15 PM, Dalia Sobhy wrote:
Dear all,
I wanted to ask a question..
Do Hbase Aggregate Functions such as rowcount, getMax, get Average
use MapReduce to execute those functions?
Thanks :D
