I have a related question (apologies in advance if I should be branching to
a new thread). I'm also doing a large amount of hadoop MR data crunching on
documents stored in hbase. I ran a simple letter count MR job on my
documents (only 150k for now, but soon to be 2.5mil) and it took a
surprisingly long time (about 9 hours). Looking at the job tracker I noticed
I only had run 4 map tasks and 1 reduce task. Given that my cluster is 10
quad-core machines I would have assumed there to be 4 map tasks on EACH
machine for a total of 40?

I'm wondering how you set up your job to run 2 maps/1 reducer per machine.
Is this a matter of adding more region servers? I currently have 1
regionserver and 144 regions (living on the same cluster as hadoop.

Thanks.

On 6/12/09, Bradford Stephens <[email protected]> wrote:
>
> Yeah, I was thinking about that. I guess data in HBase needs to be
> sorted once, so data locality is a null issue on load. So, we can have
> a separate HBase cluster and it wouldn't impact load performance any
> more than having our Hadoop data 'crunching' cluster on the same boxes
> as HBase? We plan on doing most of our MR tasks *on* documents in
> HBase, however.
>
> I'm running 2 maps and 1 reducer per machine. All the Map tasks are
> done by the time we start reducin' and loadin'.
>
> Sorry if this had too many run-on sentences :)
>
>
>
> On Fri, Jun 12, 2009 at 10:31 AM, Ryan Rawson<[email protected]> wrote:
> > yeah, sounds like it.  How many maps are you running per machine?
> >
> > You could try making the hbase and mr machines disjoint?
> >
> > On Jun 12, 2009 10:20 AM, "Bradford Stephens" <
> [email protected]>
> > wrote:
> >
> > I don't think we're using ZK, I'm on HBase-0.19.4...am I wrong? :)
> > I've already got the GC configured to do what you suggested... I'm not
> > getting very long pauses from the log file. I really think the problem
> > is resource starvation because I only have 2 total cores on each of
> > those boxes, and it's running HBase on top of Hadoop DataNodes and
> > TaskTrackers. Am I right in this thinking?
> >
> > On Thu, Jun 11, 2009 at 10:29 PM, Ryan Rawson<[email protected]> wrote:
> >
> > Since you are on a 2-4 c...
> >
>

Reply via email to