Re: Memory settings in hadoop YARN
Hi Narayanan, I've read a great blog post by Rohit Bakhshi before, recommend it to you : http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/. I think most your questions are covered by this blog post. Please let me know if you have more questions. Thanks, Wangda Tan On Wed, Aug 20, 2014 at 12:12 PM, Narayanan K knarayana...@gmail.com wrote: Hi We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture. I had few questions on memory used by the jobs : We have following settings for memory. mapred.child.java.opts mapreduce.map.memory.mb mapreduce.reduce.memory.mb yarn.app.mapreduce.am.resource.mb yarn.app.mapreduce.am.command-opts 1. I want to understand these settings to make better use of Hadoop cluster. 2. How is Memory Allocated to Container ? Does any of the above settings result in change in container ? 3. Any other memory settings we need to be aware of ? 4. I heard there was a virtual memory and physical memory involved. Is there any propery documentation/guide that can make the memory management easy. Thanks Narayanan
Memory settings in hadoop YARN
Hi We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture. I had few questions on memory used by the jobs : We have following settings for memory. mapred.child.java.opts mapreduce.map.memory.mb mapreduce.reduce.memory.mb yarn.app.mapreduce.am.resource.mb yarn.app.mapreduce.am.command-opts 1. I want to understand these settings to make better use of Hadoop cluster. 2. How is Memory Allocated to Container ? Does any of the above settings result in change in container ? 3. Any other memory settings we need to be aware of ? 4. I heard there was a virtual memory and physical memory involved. Is there any propery documentation/guide that can make the memory management easy. Thanks Narayanan
Memory settings in hadoop YARN
Hi We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture. I had few questions on memory used by the jobs : We have following settings for memory. mapred.child.java.opts mapreduce.map.memory.mb mapreduce.reduce.memory.mb yarn.app.mapreduce.am.resource.mb yarn.app.mapreduce.am.command-opts 1. I want to understand these settings to make better use of Hadoop cluster. 2. How is Memory Allocated to Container ? Does any of the above settings result in change in container ? 3. Any other memory settings we need to be aware of ? 4. I heard there was a virtual memory and physical memory involved. Is there any propery documentation/guide that can make the memory management easy. Thanks Narayanan
