Did you try reducing your spark.executor.memory? Thanks Best Regards
On Wed, Apr 15, 2015 at 2:29 PM, Brahma Reddy Battula < brahmareddy.batt...@huawei.com> wrote: > Hello Sparkers > > > I am newbie to spark and need help.. We are using spark 1.2, we are > getting the following error and executor is getting killed..I seen > SPARK-1930 and it should be in 1.2.. > > *Any pointer to following error, like what might lead this error.*. > > > 2015-04-15 11:55:39,697 | WARN | Container Monitor | Container > [pid=126843,containerID=container_1429065217137_0012_01_-411041790] is > running beyond physical memory limits. Current usage: 26.0 GB of 26 GB > physical memory used; 26.7 GB of 260 GB virtual memory used. Killing > container. > Dump of the process-tree for container_1429065217137_0012_01_-411041790 > : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) > SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE > |- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 > 6824864 /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server > -XX:OnOutOfMemoryError=kill %p -Xms24576m -Xmx24576m > -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties > -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native > -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp > -Dspark.driver.port=23204 -Dspark.random.port.max=23999 > -Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100 > -Dspark.ui.port=23000 -Dspark.random.port.min=23000 > -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 > org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp:// > sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 > 10 application_1429065217137_0012 |- 126843 76960 126843 126843 > (bash) 0 0 11603968 331 /bin/bash -c > /opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server > -XX:OnOutOfMemoryError='kill %p' -Xms24576m -Xmx24576m > -Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties > -Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native > -Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp > '-Dspark.driver.port=23204' '-Dspark.random.port.max=23999' > '-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10' > '-Dspark.akka.timeout=100' '-Dspark.ui.port=23000' > '-Dspark.random.port.min=23000' > -Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790 > org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp:// > sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3 hadoopc1h11 > 10 application_1429065217137_0012 1> > /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout > 2> > /srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr > > | > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447) > > > > > And some doubts > > > 1) why executor will not release memory, if there are tasks running..? > > > > 2) is there issue from hadoop which will lead this error..? > > > > Any help , will be appreciated... > > > > > Thanks & Regards > > Brahma Reddy Battula > > > > >
