Re: only one map or reduce job per time on one node
I suspect that the reason no-one is responding with good answers is that fundamentally, it seems like what you are trying to do runs against the reason Hadoop is designed the way it is. A parallel process framework is defeated if you force it to not work concurrently... Maybe you should look into Oozie, and set a workflow that runs your map task as a map-only job, and then do the calculation that the reducer would do in a different program? There is a Hadoop parameter that allows for task JVM reuse that might solve your problem also. Or, add a combiner to your mr job if there is not one already, and then try dropping the maximum heap sizes of the respective slots? *Devin Suiter* Jr. Data Solutions Software Engineer 100 Sandusky Street | 2nd Floor | Pittsburgh, PA 15212 Google Voice: 412-256-8556 | www.rdx.com On Wed, Nov 6, 2013 at 5:09 AM, John johnnyenglish...@gmail.com wrote: Does no one have an idea if it is possible? In my example the heapsize of the tasktracker is 2gb. The max map heap size is 1gb and the mac heap size of the reduce process 1,5gb. If there are running 2 maps or just the reduce job everything is fine. But if there are running 2 maps + 1 reduce process I got a Error: java.lang.OutOfMemoryError: Java heap space at message. kind regards 2013/11/5 John johnnyenglish...@gmail.com Because my node swaps the memory if the 2 map slots + 1 reduce is occupied with my job. Sure I can minimize the max memory for the map/reduce process. I tried this already, but I got a out of memory exception if set the max heap size for the map/reduce process to low for my mr job. kind regards 2013/11/5 Vinod Kumar Vavilapalli vino...@hortonworks.com Why do you want to do this? +Vinod On Nov 5, 2013, at 9:17 AM, John wrote: Is it possible to force the jobtracker executing only 2 map jobs or 1 reduce job per time? CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
only one map or reduce job per time on one node
hi, I have a cluster of 7 nodes. Every node has 2 maps-lots and 1 reduce slot. Is it possible to force the jobtracker executing only 2 map jobs or 1 reduce job per time? I have found this configuration option: mapred.reduce.slowstart.completed.maps. I think this will do exactly what I want If I set it to 1.0 if there is only one mapreduce job per time. But whenn there are 2 jobs I think it doesn't work because if the second job is finished with the map part it will maybe execute a reduce job on node where still the first job is running. Or am I wrong? Is there a way to allow executing reduce jobs only if there is no map reduce job on this node? kind regards
Re: only one map or reduce job per time on one node
Why do you want to do this? +Vinod On Nov 5, 2013, at 9:17 AM, John wrote: Is it possible to force the jobtracker executing only 2 map jobs or 1 reduce job per time? -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: only one map or reduce job per time on one node
Because my node swaps the memory if the 2 map slots + 1 reduce is occupied with my job. Sure I can minimize the max memory for the map/reduce process. I tried this already, but I got a out of memory exception if set the max heap size for the map/reduce process to low for my mr job. kind regards 2013/11/5 Vinod Kumar Vavilapalli vino...@hortonworks.com Why do you want to do this? +Vinod On Nov 5, 2013, at 9:17 AM, John wrote: Is it possible to force the jobtracker executing only 2 map jobs or 1 reduce job per time? CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.