y larger jobs without running into swap hell or
GC overhead limit exceeeded.
On Sun, Nov 6, 2011 at 7:11 AM, Florin P wrote:
Hello!
How many reducers you are using?
Regarding the performance parameters, fist you can increase the size of the
io.sort.mb parameter.
It seems that you are s
Hello!
How many reducers you are using?
Regarding the performance parameters, fist you can increase the size of the
io.sort.mb parameter.
It seems that you are sending a lot of amount of data to the reducer. By
increasing the value of this parameter, in the shuffle phase, the framework
w
Hello1
I would like to know how Hadoop is
computing the number of mappers when CombineFileInputFormat
is used? I have read the API specification for
CombineFileInputFormat
(http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html),
but unf
:26 AM
> yes, there is a reduce. In fact, I'm
> using hive to run map reduce
> jobs, and the reducer is a perl script.
>
> The data send to reducer is about 1/3 or 1/4 of map input
> data.
>
> On Fri, Aug 12, 2011 at 5:26 PM, Florin P
> wrote:
> > Hello!
> >
Hello!
Di you have a reducer class involved? If yes, what is the amount of data that
you are sending from the mapper to the reducer?
Regards,
Florin
--- On Fri, 8/12/11, wd wrote:
> From: wd
> Subject: some map run really slow
> To: mapreduce-user@hadoop.apache.org
> Date: Friday, August 12,
Hello!
I would like to know how Hadoop is computing the number of mappers when
CombineFileInputFormat is used? I have read the API specification for
CombineFileInputFormat
(http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html),
but unfortu
