To make sure I understand...you've allocated /ten times/ your physical
RAM for containers? If so, I think that's your issue.
For reference, under Hadoop 3.x I didn't have a cluster that would
really do anything until its worker nodes had at least 8GiB.
On 8/14/19 12:10 PM, . . wrote:
Hi all,
I installed a basic 3 nodes Hadoop 2.9.1 cluster and playing with YARN
settings.
The 3 nodes has following configuration:
1 cpu / 1 core?? / 512MB RAM
I wonder I was able to configure yarn-site.xml with following settings
(higher than node physical limits) and successfully run a mapreduce
'pi 1 10' job
quote...
?? <property>
<name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
?? ?? <property>
<name>yarn.nodemanager.resource.memory-mb</name>
?? ?? ?? ?? <value>5120</value>
?? ?? ?? ?? <description>Amount of physical memory, in MB, that can be
allocated for containers. If set to -1 and
yarn.nodemanager.resource.detect-hardware-capabilities is true, it is
automatically calculated. In other cases, the default is
8192MB</description>
?? ?? </property>
?? ?? <property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
?? ?? ?? ?? <value>6</value>
?? ?? ?? ?? <description>Number of CPU cores that can be allocated for
containers.</description>
?? ?? </property>
...unquote
Can anyone provide an explanation please?
Should 'yarn.nodemanager.vmem-check-enabled' and
'yarn.nodemanager.pmem-check-enabled' properties (set to 'true' as
default) check that my YARN settings are higher than physical limits?
Which mapreduce 'pi' job settings can I use, to 'force' containers to
use more than node physical resources?
Many thanks in advance!
Guido
