Hello Sparkers
I am newbie to spark and need help.. We are using spark 1.2, we are getting
the following error and executor is getting killed..I seen SPARK-1930 and it
should be in 1.2..
Any pointer to following error, like what might lead this error..
2015-04-15 11:55:39,697 | WARN | Container Monitor | Container
[pid=126843,containerID=container_1429065217137_0012_01_-411041790] is running
beyond physical memory limits. Current usage: 26.0 GB of 26 GB physical memory
used; 26.7 GB of 260 GB virtual memory used. Killing container.
Dump of the process-tree for container_1429065217137_0012_01_-411041790 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS)
VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 126872 126843 126843 126843 (java) 2049457 22816 28673892352 6824864
/opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError=kill
%p -Xms24576m -Xmx24576m
-Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties
-Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native
-Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp
-Dspark.driver.port=23204 -Dspark.random.port.max=23999
-Dspark.akka.threads=32 -Dspark.akka.frameSize=10 -Dspark.akka.timeout=100
-Dspark.ui.port=23000 -Dspark.random.port.min=23000
-Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790
org.apache.spark.executor.CoarseGrainedExecutorBackend
akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3
hadoopc1h11 10 application_1429065217137_0012 |- 126843 76960 126843
126843 (bash) 0 0 11603968 331 /bin/bash -c
/opt/huawei/Bigdata/jdk1.7.0_76//bin/java -server -XX:OnOutOfMemoryError='kill
%p' -Xms24576m -Xmx24576m
-Dlog4j.configuration=file:/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Spark/spark/conf/log4j-executor.properties
-Djava.library.path=/opt/huawei/Bigdata/DataSight_FM_BasePlatform_V100R001C00_Hadoop//hadoop/lib/native
-Djava.io.tmpdir=/srv/BigData/hadoop/data4/nm/localdir/usercache/ossuser/appcache/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/tmp
'-Dspark.driver.port=23204' '-Dspark.random.port.max=23999'
'-Dspark.akka.threads=32' '-Dspark.akka.frameSize=10'
'-Dspark.akka.timeout=100' '-Dspark.ui.port=23000'
'-Dspark.random.port.min=23000'
-Dspark.yarn.app.container.log.dir=/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790
org.apache.spark.executor.CoarseGrainedExecutorBackend
akka.tcp://sparkDriver@172.57.1.61:23204/user/CoarseGrainedScheduler 3
hadoopc1h11 10 application_1429065217137_0012 1>
/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stdout
2>
/srv/BigData/hadoop/data5/nm/containerlogs/application_1429065217137_0012/container_1429065217137_0012_01_-411041790/stderr
|
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:447)
And some doubts
1) why executor will not release memory, if there are tasks running..?
2) is there issue from hadoop which will lead this error..?
Any help , will be appreciated...
Thanks & Regards
Brahma Reddy Battula
