You could try increasing your heap space explicitly. like export _JAVA_OPTIONS="-Xmx10g", its not the correct approach but try.
Thanks Best Regards On Tue, Apr 28, 2015 at 10:35 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com> wrote: > I have a SparkApp that runs completes in 45 mins for 5 files (5*750MB > size) and it takes 16 executors to do so. > > I wanted to run it against 10 files of each input type (10*3 files as > there are three inputs that are transformed). [Input1 = 10*750 MB, > Input2=10*2.5GB, Input3 = 10*1.5G], Hence i used 32 executors. > > I see multiple > 5/04/28 09:23:31 WARN executor.Executor: Issue communicating with driver > in heartbeater > org.apache.spark.SparkException: Error sending message [message = > Heartbeat(22,[Lscala.Tuple2;@2e4c404a,BlockManagerId(22, > phxaishdc9dn1048.stratus.phx.ebay.com, 39505))] > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209) > at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427) > Caused by: java.util.concurrent.TimeoutException: Futures timed out after > [30 seconds] > at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) > at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) > at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107) > at > scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) > at scala.concurrent.Await$.result(package.scala:107) > at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:195) > ... 1 more > > > When i searched deeper, i found OOM error. > 15/04/28 09:10:15 INFO storage.BlockManagerMasterActor: Removing block > manager BlockManagerId(17, phxdpehdc9dn2643.stratus.phx.ebay.com, 36819) > 15/04/28 09:11:26 WARN storage.BlockManagerMasterActor: Removing > BlockManager BlockManagerId(9, phxaishdc9dn1783.stratus.phx.ebay.com, > 48304) with no recent heart beats: 121200ms exceeds 120000ms > 15/04/28 09:11:26 INFO storage.BlockManagerMasterActor: Removing block > manager BlockManagerId(9, phxaishdc9dn1783.stratus.phx.ebay.com, 48304) > 15/04/28 09:11:26 ERROR util.Utils: Uncaught exception in thread > task-result-getter-3 > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2245) > at java.util.Arrays.copyOf(Arrays.java:2219) > at java.util.ArrayList.grow(ArrayList.java:242) > at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216) > at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208) > at java.util.ArrayList.add(ArrayList.java:440) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.nextReadId(MapReferenceResolver.java:33) > at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:766) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:727) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:338) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > at > org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:173) > at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:79) > at > org.apache.spark.scheduler.TaskSetManager.handleSuccessfulTask(TaskSetManager.scala:621) > at > org.apache.spark.scheduler.TaskSchedulerImpl.handleSuccessfulTask(TaskSchedulerImpl.scala:379) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:82) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:50) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Exception in thread "task-result-getter-3" java.lang.OutOfMemoryError: > Java heap space > at java.util.Arrays.copyOf(Arrays.java:2245) > at java.util.Arrays.copyOf(Arrays.java:2219) > at java.util.ArrayList.grow(ArrayList.java:242) > at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216) > at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208) > at java.util.ArrayList.add(ArrayList.java:440) > at > com.esotericsoftware.kryo.util.MapReferenceResolver.nextReadId(MapReferenceResolver.java:33) > at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:766) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:727) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:338) > at > com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:293) > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729) > at > org.apache.spark.serializer.KryoSerializerInstance.deserialize(KryoSerializer.scala:173) > at org.apache.spark.scheduler.DirectTaskResult.value(TaskResult.scala:79) > at > org.apache.spark.scheduler.TaskSetManager.handleSuccessfulTask(TaskSetManager.scala:621) > at > org.apache.spark.scheduler.TaskSchedulerImpl.handleSuccessfulTask(TaskSchedulerImpl.scala:379) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:82) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:51) > at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1618) > at > org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:50) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > LogType: stdout > LogLength: 96 > Log Contents: > > hdfs://hostName:8020/sys/edw/dw_lstg_item/snapshot/2015/04/28/00/part-r-0000* > > > Spark Command: > > ./bin/spark-submit -v --master yarn-cluster --driver-class-path > /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-EBAY-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/yarn/lib/guava-11.0.2.jar:/apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar > --jars > /apache/hadoop-2.4.1-2.1.3.0-2-EBAY/share/hadoop/hdfs/hadoop-hdfs-2.4.1-EBAY-2.jar,/home/dvasthimal/spark1.3/1.3.1.lib/spark_reporting_dep_only-1.0-SNAPSHOT-jar-with-dependencies.jar > --num-executors 32 --driver-memory 12g --driver-java-options > "-XX:MaxPermSize=8G" --executor-memory 12g --executor-cores 4 --queue > hdmi-express --class com.ebay.ep.poc.spark.reporting.SparkApp > /home/dvasthimal/spark1.3/1.3.1.lib/spark_reporting-1.0-SNAPSHOT.jar > startDate=2015-04-6 endDate=2015-04-7 > input=/user/dvasthimal/epdatasets_small/exptsession subcommand=viewItem > output=/user/dvasthimal/epdatasets/viewItem buffersize=128 > maxbuffersize=1068 maxResultSize=200G askTimeout=1200 > > > > There is 12G limit on memory that i can use as this Spark is running over > YARN. > > Spark Version: 1.3.1 > Should i increase the number of executors form 32? > -- > Deepak > >
