I have a related question (apologies in advance if I should be branching to a new thread). I'm also doing a large amount of hadoop MR data crunching on documents stored in hbase. I ran a simple letter count MR job on my documents (only 150k for now, but soon to be 2.5mil) and it took a surprisingly long time (about 9 hours). Looking at the job tracker I noticed I only had run 4 map tasks and 1 reduce task. Given that my cluster is 10 quad-core machines I would have assumed there to be 4 map tasks on EACH machine for a total of 40?
I'm wondering how you set up your job to run 2 maps/1 reducer per machine. Is this a matter of adding more region servers? I currently have 1 regionserver and 144 regions (living on the same cluster as hadoop. Thanks. On 6/12/09, Bradford Stephens <[email protected]> wrote: > > Yeah, I was thinking about that. I guess data in HBase needs to be > sorted once, so data locality is a null issue on load. So, we can have > a separate HBase cluster and it wouldn't impact load performance any > more than having our Hadoop data 'crunching' cluster on the same boxes > as HBase? We plan on doing most of our MR tasks *on* documents in > HBase, however. > > I'm running 2 maps and 1 reducer per machine. All the Map tasks are > done by the time we start reducin' and loadin'. > > Sorry if this had too many run-on sentences :) > > > > On Fri, Jun 12, 2009 at 10:31 AM, Ryan Rawson<[email protected]> wrote: > > yeah, sounds like it. How many maps are you running per machine? > > > > You could try making the hbase and mr machines disjoint? > > > > On Jun 12, 2009 10:20 AM, "Bradford Stephens" < > [email protected]> > > wrote: > > > > I don't think we're using ZK, I'm on HBase-0.19.4...am I wrong? :) > > I've already got the GC configured to do what you suggested... I'm not > > getting very long pauses from the log file. I really think the problem > > is resource starvation because I only have 2 total cores on each of > > those boxes, and it's running HBase on top of Hadoop DataNodes and > > TaskTrackers. Am I right in this thinking? > > > > On Thu, Jun 11, 2009 at 10:29 PM, Ryan Rawson<[email protected]> wrote: > > > > Since you are on a 2-4 c... > > >
