I suspect that the reason no-one is responding with good answers is that fundamentally, it seems like what you are trying to do runs against the reason Hadoop is designed the way it is. A parallel process framework is defeated if you force it to not work concurrently...
Maybe you should look into Oozie, and set a workflow that runs your map task as a map-only job, and then do the calculation that the reducer would do in a different program? There is a Hadoop parameter that allows for task JVM reuse that might solve your problem also. Or, add a combiner to your mr job if there is not one already, and then try dropping the maximum heap sizes of the respective slots? *Devin Suiter* Jr. Data Solutions Software Engineer 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 Google Voice: 412-256-8556 | www.rdx.com On Wed, Nov 6, 2013 at 5:09 AM, John <johnnyenglish...@gmail.com> wrote: > Does no one have an idea if it is possible? In my example the heapsize of > the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size > of the reduce process 1,5gb. If there are running 2 maps or just the reduce > job everything is fine. But if there are running 2 maps + 1 reduce process > I got a "Error: java.lang.OutOfMemoryError: Java heap space at" message. > > kind regards > > > 2013/11/5 John <johnnyenglish...@gmail.com> > >> Because my node swaps the memory if the 2 map slots + 1 reduce is >> occupied with my job. Sure I can minimize the max memory for the map/reduce >> process. I tried this already, but I got a out of memory exception if set >> the max heap size for the map/reduce process to low for my mr job. >> >> kind regards >> >> >> 2013/11/5 Vinod Kumar Vavilapalli <vino...@hortonworks.com> >> >>> Why do you want to do this? >>> >>> +Vinod >>> >>> On Nov 5, 2013, at 9:17 AM, John wrote: >>> >>> Is it possible to force the jobtracker executing only 2 map jobs or 1 >>> reduce job per time? >>> >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >> >> >> >
