All this means is that your JVM is using more memory than it requested from YARN. You need to increase the YARN memory overhead setting, perhaps.
On Wed, Apr 15, 2015 at 9:59 AM, Brahma Reddy Battula <brahmareddy.batt...@huawei.com> wrote: > Hello Sparkers > > > I am newbie to spark and need help.. We are using spark 1.2, we are getting > the following error and executor is getting killed..I seen SPARK-1930 and it > should be in 1.2.. > > Any pointer to following error, like what might lead this error.. > > > 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container > [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is > running beyond physical memory limits. Current usage: 26.0 GB of 26 GB > physical memory used; 26.7 GB of 260 GB virtual memory used. Killing > container. > Dump of the process-tree for container_1429065217137_0012_01_-411041790 : > |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 > 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server > -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m > -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties > -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native > -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp > -Dspark.driver.port=23204 -Dspark.random.port.max=23999 > -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 > -Dspark.ui.port=23000 -Dspark.random.port.min=23000 > -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 > org.apache.spark.executor.CoarseGrainedExecutorBackend > akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 > hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843 > 126843 (bash) 0 0 11603968 331 /bin/bash -c > /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server > -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m > -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties > -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native > -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp > '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' > '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' > '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' > '-Dspark.random.port.min=23000' > -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 > org.apache.spark.executor.CoarseGrainedExecutorBackend > akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 > hadoopc1h11 10 application_1429065217137_0012 1> > /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout > 2> > /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr > | > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) > > > > And some doubts > > > 1) why executor will not release memory, if there are tasks running..? > > > > 2) is there issue from hadoop which will lead this error..? > > > > Any help , will be appreciated... > > > > > Thanks & Regards > > Brahma Reddy Battula > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
