Hi All,
I’m currently running Hadoop 2.7.2 on my three node cluster, all node equiped
with 32-core CPU and 64 GB memory, Ubuntu 14.04.3 LTS. Yarn’s configuration
kept the same with default one, except yarn.nodemanager.resource.memory-mb set
as 16384(16G).
Everything worked fine, but when I try to run Map/Reduce task with JVM options
mapreduce.map.java.opts=-XX:+PrintGCDetails, the task failed, with error
message said that container is running beyond virtual memory limits. Current
usage: 212.0 MB of 1 GB physical memory used; 17.9 GB of 2.1 GB virtual memory
used. Killing container.
Actually I was just running the Map/Reduce expample pi with tiny calculation,
hadoop jar hadoop-mapreduce-examples-2.7.2.jar pi -D
mapreduce.map.java.opts=-XX:+PrintGCDetails 5 100.
I’ve tried to disable yarn.nodemanager.vmem-check-enabled, but as calculation
increasees, error happens again.
I am really confused why huge virtual memory allcated when JVM option
-XX:+PrintGCDetails was added. Please help!
Thank You,
JR
Error Message
16/04/29 09:43:47 INFO mapreduce.Job: Task Id :
attempt_1461846504106_0012_m_000004_1, Status : FAILED
Container [pid=16629,containerID=container_1461846504106_0012_01_000010] is
running beyond virtual memory limits. Current usage: 212.0 MB of 1 GB physical
memory used; 17.9 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1461846504106_0012_01_000010 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS)
VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 16629 16627 16629 16629 (bash) 0 0 17051648 667 /bin/bash -c
/usr/lib/jvm/jdk8/bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails
-Djava.io.tmpdir=/opt/hadoop-2.7.2/tmp/nodemanager/local/usercache/hadoop/appcache/application_1461846504106_0012/container_1461846504106_0012_01_000010/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.13.71.82
56755 attempt_1461846504106_0012_m_000004_1 10
1>/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010/stdout
2>/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010/stderr
|- 16633 16629 16629 16629 (java) 399 25 19239059456 53616
/usr/lib/jvm/jdk8/bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails
-Djava.io.tmpdir=/opt/hadoop-2.7.2/tmp/nodemanager/local/usercache/hadoop/appcache/application_1461846504106_0012/container_1461846504106_0012_01_000010/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/opt/hadoop-2.7.2/tmp/nodemanager/logs/application_1461846504106_0012/container_1461846504106_0012_01_000010
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.13.71.82
56755 attempt_1461846504106_0012_m_000004_1 10
yarn-site.xml
<configuration><!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>namenode:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>namenode:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>namenode:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>namenode:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>namenode:8088</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>${hadoop.tmp.dir}/nodemanager/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>${hadoop.tmp.dir}/nodemanager/logs</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
</property>
<property>
<!-- Where to aggregate logs to. -->
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>${hadoop.tmp.dir}/nodemanager/remote</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
<property>
<!-- enable log aggregation -->
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<!-- How long to keep aggregation logs before deleting them. -1
disables. -->
<name>yarn.log-aggregation.retain-seconds</name>
<!-- 30 days -->
<value>2592000</value>
</property>
<property>
<!-- How long to wait between aggregated log retention checks. If set
to 0 or a negative value then the value is computed as one-tenth of the
aggregated log retention time. -->
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>-1</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>16384</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>true</value>
</property></configuration>
