Explanation here. http://stackoverflow.com/questions/24070557/what-is-the-relation-between-mapreduce-map-memory-mb-and-mapred-map-child-jav https://support.pivotal.io/hc/en-us/articles/201462036-Mapreduce-YARN-Memory-Parameters http://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-common/ClusterSetup.html (scroll towards the end.)
Regards, Shahab On Wed, Oct 15, 2014 at 9:24 AM, SACHINGUPTA <sac...@datametica.com> wrote: > I have one more doubt i was reading this > > > http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html > > there is one property as > > mapreduce.map.memory.mb = 2*1024 MB > mapreduce.reduce.memory.mb = 2 * 2 = 4*1024 MB > what are these properties mapreduce.map.memory.mb and > mapreduce.reduce.memory.mb > > On Wednesday 15 October 2014 06:17 PM, Shahab Yunus wrote: > > It cannot run more mappers (tasks) in parallel than the underlying cores > available. Just like it cannot run multiple mappers in parallel if each > mapper's (task's) memory requirements are greater than allocated and > available container size configured on each node. > > The links that I provided earlier...see the following section in that > one: > Section:"Configuring YARN" > > Also this: > http://blog.cloudera.com/blog/2014/04/apache-hadoop-yarn-avoiding-6-time-consuming-gotchas/ > Section "1. YARN Concurrency (aka “What Happened to Slots?”)" > > This should help in putting things in perspective regarding how resource > allocation for each task, container and resources available on the node > relate to each other. > > Regards, > Shahab > > On Wed, Oct 15, 2014 at 8:18 AM, SACHINGUPTA <sac...@datametica.com> > wrote: > >> but Shahab if i have only 4 core machine then how yarn can run more then >> 4 mappers in parallel >> On Wednesday 15 October 2014 05:45 PM, Shahab Yunus wrote: >> >> It depends on memory settings as well, that how much you want to assign >> resources to each container. Then yarn will run as many mappers in parallel >> as possible. >> >> See this: >> http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/ >> >> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html >> >> Regards, >> Shahab >> >> On Wed, Oct 15, 2014 at 8:09 AM, SACHINGUPTA <sac...@datametica.com> >> wrote: >> >>> Hi guys >>> >>> I have situation in which i have machine with 4 processor and i have 5 >>> containers so does it mean i can have only 4 mappers running parallely at a >>> time >>> >>> and number of mappers is not dependent on the number of containers in a >>> machine then what is the use of container concept >>> >>> sorry if i have asked anything obvious. >>> >>> -- >>> Thanks >>> Sachin Gupta >>> >>> >> >> -- >> Thanks >> Sachin Gupta >> >> > > -- > Thanks > Sachin Gupta > >
