Jamal,
This is what I am using...
After you start your job, visit jobtracker's WebUI :50030
And look for Cluster summary. Reduce Task Capacity shall hint you what
optimally set your number to. I could be wrong but it works for me. :)
Cluster Summary (Heap Size is *** MB/966.69 MB)
Running Map Ta
Hi Sasha
In general the number of reduce tasks is chosen mainly based on the data volume
to reduce phase. In tools like hive and pig by default for every 1GB of map
output there will be a reducer. So if you have 100 gigs of map output then 100
reducers.
If your tasks are more CPU intensive then
s - 6:5mins
16 Reducers - 5.5mins
8 Reducers have won the race. But Reducers at the max capacity was very clos. :)
AK47
From: Bejoy KS [mailto:bejoy.had...@gmail.com]
Sent: Wednesday, November 21, 2012 11:51 AM
To: user@hadoop.apache.org
Subject: Re: guessing number of reducers.
Hi Sasha
In genera
t;
> 10 Reducers – 7mins
>
> 12 Reducers – 6:5mins
>
> 16 Reducers – 5.5mins
>
>
>
> 8 Reducers have won the race. But Reducers at the max capacity was very
> clos. J
>
>
>
> AK47
>
>
>
>
>
> *From:* Bejoy KS [mailto:bejoy.had...@gmail.co
have won the race. But Reducers at the max capacity was very
> clos. J
>
>
>
> AK47
>
>
>
>
>
> *From:* Bejoy KS [mailto:bejoy.had...@gmail.com]
> *Sent:* Wednesday, November 21, 2012 11:51 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: guessing number of reducers.
&g
KS
Sent from handheld, please excuse typos.
-Original Message-
From: "Kartashov, Andy"
Date: Wed, 21 Nov 2012 17:49:50
To: user@hadoop.apache.org;
bejoy.had...@gmail.com
Subject: RE: guessing number of reducers.
Bejoy,
I've read somethere about keeping number of mapr
:50 +
> To: user@hadoop.apache.org; bejoy.had...@gmail.com
> Subject: RE: guessing number of reducers.
>
> Bejoy,
>
>
>
> I’ve read somethere about keeping number of mapred.reduce.tasks below the
reduce task capcity. Here is what I just tested:
>
>
>
> Output 25Gb.
.
You can round this value and use it to set the number of reducers in conf
programatically.
Regards
Bejoy KS
Sent from handheld, please excuse typos.
-Original Message-
From: Manoj Babu
Date: Wed, 21 Nov 2012 23:28:00
To:
Cc: bejoy.had...@gmail.com
Subject: Re: guessing number of
gt;
> Sent from handheld, please excuse typos.
> --
> *From: * Manoj Babu
> *Date: *Wed, 21 Nov 2012 23:28:00 +0530
> *To: *
> *Cc: *bejoy.had...@gmail.com
> *Subject: *Re: guessing number of reducers.
>
> Hi,
>
> How to set no of reducers in job conf dynami
