Hi All,
I have a program which actually run a bit complex business (join) in spark. And I have below exception: I running on Spark 1.5, and with parameter: spark-submit --deploy-mode client --executor-cores=24 --driver-memory=2G --executor-memory=45G -class . Some other setup: sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer").set("spark.kryoserializer.buff er.max", "2047m"); sparkConf.set("spark.executor.extraJavaOptions", "-XX:+PrintGCDetails -XX:+PrintGCTimeStamps").set("spark.sql.autoBroadcastJoinThreshold", "104857600"); This is running on AWS c3*8xlarge instance. I am not sure what kind of parameter I should set if I have below OutOfMemoryError exception. # # java.lang.OutOfMemoryError: Java heap space # -XX:OnOutOfMemoryError="kill -9 %p" # Executing /bin/sh -c "kill -9 10181"... Exception in thread "broadcast-hash-join-2" java.lang.OutOfMemoryError: Java heap space at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProje ction.apply(Unknown Source) at org.apache.spark.sql.execution.joins.UnsafeHashedRelation$.apply(HashedRelat ion.scala:380) at org.apache.spark.sql.execution.joins.HashedRelation$.apply(HashedRelation.sc ala:123) at org.apache.spark.sql.execution.joins.BroadcastHashOuterJoin$$anonfun$broadca stFuture$1$$anonfun$apply$1.apply(BroadcastHashOuterJoin.scala:95) at org.apache.spark.sql.execution.joins.BroadcastHashOuterJoin$$anonfun$broadca stFuture$1$$anonfun$apply$1.apply(BroadcastHashOuterJoin.scala:85) at org.apache.spark.sql.execution.SQLExecution$.withExecutionId(SQLExecution.sc ala:100) at org.apache.spark.sql.execution.joins.BroadcastHashOuterJoin$$anonfun$broadca stFuture$1.apply(BroadcastHashOuterJoin.scala:85) at org.apache.spark.sql.execution.joins.BroadcastHashOuterJoin$$anonfun$broadca stFuture$1.apply(BroadcastHashOuterJoin.scala:85) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future. scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 45) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 15) at java.lang.Thread.run(Thread.java:745) Any hint will be very helpful. Regards, Shuai