RE: How to configure multiple reduce jobs in hadoop 2.2.0

gs? From what I thought, should one be enough? 2) For the above settings, my understanding is that from application, the max memory I can use for mapper task is 3G, no matter what I asked for, right? Is the additional 1G meaning any size I can ask outside of the JVM Heap? Thanks Yong

RE: How to configure multiple reduce jobs in hadoop 2.2.0

matter what I asked for, right? Is the additional 1G meaning any size I can ask outside of the JVM Heap? Thanks Yong Date: Fri, 17 Jan 2014 15:16:28 +0530 Subject: Re: How to configure multiple reduce jobs in hadoop 2.2.0 From: sudhakara...@gmail.com To: user@hadoop.apache.org Also check th

Re: How to configure multiple reduce jobs in hadoop 2.2.0

Also check this http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/ On Fri, Jan 17, 2014 at 2:56 PM, Silvina Caíno Lores wrote: > Also, you should be limited by your container configuration at > yarn-site.xml and mapred-site.xml, check > THIS

Re: How to configure multiple reduce jobs in hadoop 2.2.0

Also, you should be limited by your container configuration at yarn-site.xml and mapred-site.xml, check THISto understand how resource management works. Basically you can set the number of reducers you want but you are limited

Re: How to configure multiple reduce jobs in hadoop 2.2.0

Hello Ashish, Using “-D mapreduce.job.reduces=number” with fixed number of reducer will spawn that many for a job. On Thu, Jan 16, 2014 at 12:45 PM, Ashish Jain wrote: > Dear All, > > I have a 3 node cluster and have a map reduce job running on it. I have 8 > data blocks spread across all the

Re: How to configure multiple reduce jobs in hadoop 2.2.0

Yes, you need to manually set it. Please see the API at http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/mapred/JobConf.html#setNumReduceTasks(int) On Thu, Jan 16, 2014 at 12:45 PM, Ashish Jain wrote: > Dear All, > > I have a 3 node cluster and have a map reduce job running on it. I ha