Correct: I set 'yarn.nodemanager.resource.memory-mb' ten times the node physical memory (512MB) and I was able to successfully execute a 'pi 1 10' mapreduce job.
Since default 'yarn.app.mapreduce.am.resource.mb' value is 1536MB I expected the job to never start / be allocated and I have no valid explanation. On Wed, Aug 14, 2019 at 10:32 PM . . <writeme...@googlemail.com> wrote: > Correct: I set 'yarn.nodemanager.resource.memory-mb' ten times the node > physical memory (512MB) and I was able to successfully execute a 'pi 1 10' > mapreduce job. > > Since default 'yarn.app.mapreduce.am.resource.mb' value is 1536MB I > expected the job to never start / be allocated and I have no valid > explanation. > > > > On Wed, Aug 14, 2019 at 8:31 PM Jeff Hubbs <jhubbsl...@att.net> wrote: > >> To make sure I understand...you've allocated *ten times* your physical >> RAM for containers? If so, I think that's your issue. >> >> For reference, under Hadoop 3.x I didn't have a cluster that would really >> do anything until its worker nodes had at least 8GiB. >> >> On 8/14/19 12:10 PM, . . wrote: >> >> Hi all, >> >> I installed a basic 3 nodes Hadoop 2.9.1 cluster and playing with YARN >> settings. >> The 3 nodes has following configuration: >> 1 cpu / 1 core?? / 512MB RAM >> >> I wonder I was able to configure yarn-site.xml with following settings >> (higher than node physical limits) and successfully run a mapreduce 'pi 1 >> 10' job >> >> quote... >> ?? <property> >> ?? ?? ?? >> <name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> >> </property> >> >> ?? ?? <property> >> ?? ?? ?? ?? <name>yarn.nodemanager.resource.memory-mb</name> >> ?? ?? ?? ?? <value>5120</value> >> ?? ?? ?? ?? <description>Amount of physical memory, in MB, that can be >> allocated for containers. If set to -1 and >> yarn.nodemanager.resource.detect-hardware-capabilities is true, it is >> automatically calculated. In other cases, the default is >> 8192MB</description> >> ?? ?? </property> >> >> ?? ?? <property> >> ?? ?? ?? ?? <name>yarn.nodemanager.resource.cpu-vcores</name> >> ?? ?? ?? ?? <value>6</value> >> ?? ?? ?? ?? <description>Number of CPU cores that can be allocated for >> containers.</description> >> ?? ?? </property> >> ...unquote >> >> Can anyone provide an explanation please? >> >> Should 'yarn.nodemanager.vmem-check-enabled' and >> 'yarn.nodemanager.pmem-check-enabled' properties (set to 'true' as default) >> check that my YARN settings are higher than physical limits? >> >> Which mapreduce 'pi' job settings can I use, to 'force' containers to use >> more than node physical resources? >> >> Many thanks in advance! >> Guido >> >> >>
